[simonh]

> Why is inserting the best outputs from GPT-4 into the training set for GPT-5 expected to make things worse?

Firstly what makes you think only the best output from 4 will go into future training sets? It’s just as likely to be the most bizarre or ludicrous, or dangerous that gets shared and discussed.

But also, how will v5 get to be better than v4 if it’s trained significantly on v4 output? It would just end up being trained to be the same, to have the same flaws and quirks reinforced.

We already know v4 just makes stuff up, it’s incredibly good at producing well formatted plausible looking but utterly factually incorrect output. That’s because it has no concept of truth or facts. All it knows about from the token sequence weightings is the form of language, not the content. Feeding that back into future models is the last thing we should be doing.

[sebzim4500]

>Firstly what makes you think only the best output from 4 will go into future training sets? It’s just as likely to be the most bizarre or ludicrous

That's true now, because LLMs are new so the failure cases are still interesting. If we are talking about a hypothetical world in which LLM outputs are a significant portion of the internet, then most of it would be from reddit comments/tweets/HN posts/buzzfeed articles/etc.

Then if you take only the ones which have more than average views/upvotes/etc. you should expect to get the 'best' results.

[simonh]

I'm still not convinced that's a reliable indicator of quality. It's potentially a measure of popularity or entertainment value, or maybe pandering to preconceptions but that's not at all the same thing.

Ask yourself, what are your from-scratch metrics for quality that you would like to select for. Then consider what are the likely or possible criteria people actually have for upvoting stuff on reddit. I'll think you'll find there is probably very little correlation between those. This is called the alignment problem and it's very hard to get right.