[avidphantasm]

Very little of the “AI” boom has been novel, most has been iterative elaborations (though innovative nonetheless). Academics have been using neural network statistical models for decades. What’s new is the combination of compute capability and data volume available for training. It’s iterative all the way down though, that’s how all technologies are developed.

[sigmoid10]

Most people don't realize this, but almost all research works that way. Only the media spins research as breakthrough-based, because that way it is easier to sell stories. But almost everything is incremental/iterative. Even the transformer architecture, which in some way can be seen as the most significant architectural advancement in AI in the past years, was a pretty small, incremental step when it came out. Only with a lot of further work building on top of that did it become what we see today. The problem is that science-journalists vastly outnumber scientists producing these incremental steps, so instead of reporting on topics when improvements actually accumulated to a big advancement, every step along the way gets its own article with tons of unnecessary commentary heralding its features.

[astrange]

The "bitter lesson of machine learning" means that you actually can't do anything novel; it won't work as well as just doing the simple thing but bigger.

(So there is room left if you're limited by memory or budget.)

[w10-1]

> What’s new is the combination of compute capability and data volume available for training

This is the important part.

My advisor said new means old method applied to new data or new method on old data.

Commercially, that means price points, i.e., discrete points where something becomes viable.

Maybe that's iterative, but maybe not. Either way, once the opportunity presents, time is of the essence.