[hunterpayne]

"The software is the sum total of collective human culture they are trained on."

Almost, they are the median or most popular aspects of the culture upon which they are trained. So you are getting the most popular way to do something, not the best (for some definition of best). That's why the claims about LLMs being geniuses is absurd. They almost by definition are going to have the average IQ of all the people on the net weighted by how much each person posts. I'm guessing that's about 95.

[K0balt]

Meh, while I’d agree that LLMs are idiot savants more than geniuses, I think you underestimate the general quality of training data. First, it’s all on data that was published or written. People below 80 is don’t publish or write at all, and when they do you can filter it with a regex. So already you skew the curve up 15 points or so. Then, factor in that published usually means 120+ and also includes the collective treasures of civilization. Even the average joes are going to skew towards things they are knowledgeable and passionate about, putting their best foot forward and so on. ( and the trolls get regexed to oblivion). Only the very clever trolls get through, and at least they pattern match for clever.