[aaroninsf]

True, and relevant (I live with a professional editor)... yet I immediately think of Ximm's Law:

Every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon.

Lemma: any statement about AI which uses the word "never" to preclude some feature from future realization is false.

Lemma: contemporary implementations have already improved; they're just unevenly distributed.

[edude03]

I think it can't be improved because it's measuring the wrong thing. A junior engineer becomes a senior when they stop being told what code to write and start solving business needs. Therefore often the highest paid engineers aren't the ones who would do the best on leetcode - or SWE bench pro verified.

Maybe AGI is possible and we'll have software defined human intelligence that's completely autonomous but that's not coming in the next slightly better RL trained LLM and if existed likely wouldn't be under our control anyway

[Planktonne]

No one assumes that AI systems won't be improved upon. What people don't assume is that progress will be infinite in every domain cheaply forever.

[aaroninsf]

People don't make this assumption, if you ask (or reflect);

arguments however regularly are implicitly predicated on exactly this premise.

(Infinite improvement, no; enough improvement that practically speaking the difference between effectively free and free is meaningless—that may well be the ultimate bitter lesson.)