I don't know where this idea comes from that we can get more from language models then what we put inside. Thinking we can process any amount of data and get a competent surrogate mind out of it borders on magical thinking.
>Getting out what we put in is kind of par for the course in education, yes?
Yes, you put in a person + knowledge and get out an educated person. Is it reasonable to expect to put in GPU + text and get somehow a competent actor, however narrowly we define competence (maybe if we define actor narrowly enough)?