[aspenmartin]

> To this you can add the specific testable training that pertains to code, but this degrades the weights for more general communication.

Im not sure exactly what you’re saying here, is it that you trade off coding performance and performance on other tasks like communication? If so (correct if not) this isn’t true — generalization happens. Doing good on coding lifts all boats.

> Somehow the hype over the successes with coding in the last year or so made everyone forget the intrinsic limit posed by the exhaustion of real human text output, which is absolutely inescapable

You’re absolutely right that we’re quickly running out of human text data but that isn’t at all the limitation you think it is. No one has “forgotten” this — coding agent performance is primarily from reinforcement learning on synthetic data traces with verifiable rewards, though pretraining is still important.

Also don’t forget: there is a world of multimodal data (video, audio, 3D maps, etc) that is incredibly rich.