[nb3423]

".. it must ever be a parasite on human cultural production for it to gain what you call "knowledge"

This thing goes way deep into AI theories, but probaly AGI and artificial beings self-consciusness are two different problems. I think when the second problem gets solved, the human cultural production would no longer be necessary to "create" new AI / AGIs.

[pnf]

Yes, I think that is one idea. But without true symbolism, where is the ground of understanding that would allow for synthetic generation going to come from? Right now, it has to be added by human coaches and means nothing to the model. Personally, I'm not confident we're anywhere near solving for artificial self-consciousness. Heck, we don't even understand human consciousness. But forever is a long time. I'm both impressed and unimpressed by the current LLM models. They will get better at what they do, I'm confident of that. But it does seem there's built-in limitation that can't be solved with this approach. It will be interesting, for instance, if logic gates can be encoded in a neural net architecture. That could allow them to reason the way a computer using a formal language today might. In that future, the language models are massively effective parsers plus some layers enabling it to be "Turing complete".