[ACCount37]

The biggest issue is that LLMs can act like a person indeed. And the kind of person an LLM acts as? Well.

When you train on raw user feedback, you can easily end up wiring some incredibly undesirable patterns into your AI. Resulting in things like an AI that never wants to contradict its user, and always wants to support its user in everything, and always wants the user to like it. See GPT-4o for the kind of outcomes that results in.