[weavejester]

"I can tell you, "if something's fishy about the situation, just leave". You can do this not because of some particular training inputs or examples I give you, but because you have common sense and a sort of personality and intuition for how to behave in the absence of data."

Only if I had a baseline to compare the situation to. If you took out all familiar elements, I'd have no way of telling whether a situation was normal or suspicious. My understanding of the word "fishy" is born from 300 thousand hours of training data.

"It's mostly because we have no idea how to design them anyway else. I think if anyone knew how to build complex agents with rich internal states that have the intent and communication abilities of humans we'd do that."

That's a different question. We can build machines that learn autonomously; they just don't have the capability of biological minds.