[PollardsRho]

The time span on which these developments take place matter a lot for whether the bitter lesson is relevant to a particular AI deployment. The best AI models of the future will not have 100K lines of hand-coded edge cases, and developing those to make the models of today better won't be a long-term way to move towards better AI.

On the other hand, most companies don't have unlimited time to wait for improvements on the core AI side of things, and even so building competitive advantages like a large existing customer base or really good private data sets to train next-gen AI tools have huge long-term benefits.

There's been an extraordinary amount of labor hours put into developing games that could run, through whatever tricks were necessary, on whatever hardware actually existed for consumers at the time the developers were working. Many of those tricks are no longer necessary, and clearly the way to high-definition real-time graphics was not in stacking 20 years of tricks onto 2000-era hardware. I don't think anyone working on that stuff actually thought that was going to happen, though. Many of the companies dominating the gaming industry now are the ones that built up brands and customers and experience in all of the other aspects of the industry, making sure that when better underlying scaling came there they had the experience, revenue, and know-how to make use of that tooling more effectively.

[spongebobstoes]

Why must the best model not have 100k edge cases hand coded?

Our firsthand experiences as humans can be viewed as such. People constantly over index on their own anecdata, and are the best "models" so far.

[PollardsRho]

Previous experience isn't manual edge cases, it's training data. Humans have incredible scale (100 trillion synapses): we're incredibly good at generalizing, e.g., how to pick up objects we've never seen before or understanding new social situations.

If you want to learn how to play chess, understanding the basic principles of the game is far more effective than trying to memorize every time you make an opening mistake. You surely need some amount of rote knowledge, but learning how to appraise new chess positions scales much, much better than trying to learn an astronomically small fraction of chess positions by heart.

[wegfawefgawefg]

Actually companies can just wait. Multiple times my company has said: "a new model that solves this will probably come out in like 2-4 months anyways, just leave the old one as is for now".

It has been true like ten times in the past two years.

[PollardsRho]

It's not that technical work is guaranteed to be in your codebase 10 years from now, it's that customers don't want to use a product that might be good six months from now. The actors in the best position to use new AI advances are the ones with good brands, customer bases, engineering know-how that does transfer, etc.

[wegfawefgawefg]

"those who have more capital have an advantage"