[skjoldr]

Intent and self-awareness do come from experience.

The goals we want to achieve are tied to what has been going on throughout our lives thus far. We don't invent our own values out of a vacuum, we tend to adopt them from other people, and rank them depending on how our nervous system works.

Self-awareness is the ability to reason about the process of reasoning, plus, in your example, a recollection of past examples of incorrect reasoning that disrupted the process of achieving our goals.

So basically current models need a memory and the ability to change their own weights in response to new data to even start approximating humans, but that looks like a doable task, in principle.