The only reason to believe that statement would be that training data is finite and cannot be meaningfully synthetically generated in a way that is useful to the LLM model.
If you can agree that there are certain things which can be qualitatively measured by deterministic logic (e.g. "does this build", "what is the cyclomatic complexity of this", "does this pass the unit tests", "what is the performance characteristic of this", "can this be proven to be susceptible to a XSS bug", ...), and you can see that there are ways to use this information for feedback into the models, then there's no reason to think that the available training data is finite and limited by unclean generated data.
There's several missing steps in that logic that would be difficult to (linguistically) prove with certainty, but I'm reasonably sure that your statement is false.
The core of my argument was against "There is no reason to think that LLMs will improve at all." This is a falsifiable statement if there is one or more reasons to think that LLMs will improve. I provided one such reason. I don't disagree with the part of the statement that "They may degrade to due lack of clean training data." as instances of this have already, and will in future happen. However this is immaterial to the totality of the statement.
Your argument is that there are some reasons to agree with the statement. To show that my argument is false you actually need to show that there are no reasons to disagree with the statement. In effect you're attempting to argue that because you saw some red cars means that another person's statement that all cars are red is true.
Meta argument aside, there are many other reasons to suggest that LLMs will continue to improve, the easiest of which is they have done so recently so far.
Synthetic data is doing wonders for models like Phi-4, and at least part of the dataset for DeepSeek-R1 came from their earlier models.
If you read the literature from the Phi-4 team it talks about synthetic data allowing better control over the training process. The upfront investment is greater but pays off over multiple generations of trained models - and doesn’t leave you with SolidGoldMagikarp ;)
I've rarely seen this, mostly get answers correctly, and am able to use it for generation and reasoning. Can you share any link of a chatgpt/gemini conversation where this kind of circular conversation happened?
If you can agree that there are certain things which can be qualitatively measured by deterministic logic (e.g. "does this build", "what is the cyclomatic complexity of this", "does this pass the unit tests", "what is the performance characteristic of this", "can this be proven to be susceptible to a XSS bug", ...), and you can see that there are ways to use this information for feedback into the models, then there's no reason to think that the available training data is finite and limited by unclean generated data.
There's several missing steps in that logic that would be difficult to (linguistically) prove with certainty, but I'm reasonably sure that your statement is false.