[bonoboTP]

I guess one can argue it's a novel form of (secular) divination, like tarot, except you have no idea what sort of intentions have shaped the AI during fine tuning, and by trusting it too much, you basically get puppeteered subtly by these companies.

Despite progress in natural sciences and tech, the answer to what's the good life is still up for judgment and values, it's not some neutral factual thing that simply falls out of AI training. Without fine tuning, supervised and RLHF, it could just as much role play a deranged psycho giving out life advice. To the extent that it isn't doing that, it's just adhering to whatever the company thought will land them in the least (legal and media) trouble. They had no way of actually tying it to life outcomes. It's more about what the user will like best in the moment. Which is a lot of affirmation and glazing. This is not unique to AI though, self help books can have the same failure mode especially when picked based on what feels nice to read.