[wegfawefgawefg]

If you knew everyone would ask gpt before doing anything, you would make gpt say what woudl generally be considered the better option. Not going to war, not committing suicide, etc. In this way even if war was the optimal decision according to some other utility function, the behavior of people is directed in a positive way. (Presumably)

[throwanem]

Sure, if you also assume people follow whatever advice so given. They won't, even before the covert influence effort becomes popular knowledge, as it inevitably will. This destroys consumer trust in your product after you have successfully made that product indispensable, thus opening up a previously impossible vacuum in epistemology and thus access to power.

[wegfawefgawefg]

For the record I believe it to be immoral to manipulate humanity in this way. And I also believe it might be bad for bussiness.

I was just trying to explain to the guy above what I think the guy above that meant.

[throwanem]

'The guy above' is also me :) And yeah, I get it. I guess the thing I'm trying to get at by extension and example is just how hard a problem this is, and maybe also that the formulation given assumes LLMs have a level of control over human behavior that not only doesn't exist but in the general case sort of can't since the LLM's user is always free not to take its advice. (At the very least, if humans have no option but to follow LLM instructions, something has gone much more badly wrong than the risk of there being poor instructions...)

In general, I think it's a good example of the kind of social problem tech can make a lot worse but no better: when a society has lost its grasp of epistemology, multiplying the amount of information available, at a net decrease in quality and reliability, merely multiplies the scope of the problem.