[mmoskal]

The bitter lesson is about the balance of human ingenuity and compute being thrown at the problem. We've seen a few years of LLM compute being scaled up 10x every year, but this is hitting limits (fabs), and we will see more human effort, as it becomes compartively cheaper.

Also the current crop of models are inherently limited. Even for something as simple as following a JSON schema, models alone are not good enough [0]

Of course as the Moore law refuses to die, we'll continue seeing 1.5-2x or so every year, but that's far from 10x.

[0] https://openai.com/index/introducing-structured-outputs-in-t... - see plot

[noosphr]

>Of course as the Moore law refuses to die, we'll continue seeing 1.5-2x or so every year, but that's far from 10x.

This is another one of those anec-data throw away sentences that take thousands of words to disprove - with a lot of graphs - that no one reads.

More hot takes: Moores law has been effectively dead since the Pentium 4 on CPUs. It's been dead on GPUs since 2020. Right now we're not seeing a 1.5-2x grow of compute per year. We've seen zero growth for 5. The only way that GPUs have gotten faster is by running ever hotter, and building out a trillion dollars worth of data centers.

No one cares because the current hotness in AI is transformers which are memory bound in both training and inference. If someone manages to get diffusion models to become the next hotness all of a sudden everyone will realize this is a problem since those are compute bound by a huge margin and current gen GPUs are fire hazards when ran at 100% utilization for weeks on end.

[fiddlerwoaroof]

Yeah, my experience is that the rapid growth of computing capabilities of my childhood more or less ended in 2013: the desktop I built then is fine for most of my uses and the significant performance improvements since then haven’t been CPUs getting noticeably faster but increases in the speed of disks (SSDs) and growth of memory sizes and network speeds.

[imtringued]

>More hot takes: Moores law has been effectively dead since the Pentium 4 on CPUs.

AMD is offering double digit percentage increases in IPC every generation. What you were saying might have been valid in early 2017, back when Intel was cranking up the clock speeds for a 5% increase, but when Zen came out suddenly AMD started delivering again.

[calvinmorrison]

the only escape from the Black Pill of AI ruining society is we hit some hard limits and technology stops progressing at such a rapid rate, tapers, and gives humanity a bit of time to adapt culturally.

[wegfawefgawefg]

wat... my 9950x cpu that i just bought is way faster than the similarly priced cpu i bought 6 years ago.

The difference is night and day. What are you talking about.

[adgjlsfhk1]

the equivalent chip 6 years ago was the 3950x which is the same number of cores clocked ~20% slower. if you add on ~30% ipc (for avx-512 and some general cleanups), you get a 60% speedup in 6 years which is way below the ~10x speedup between the similarly spaced Pentium 4 and Sandy bridge

[cma]

Its closer to 2X faster on Chromium compile benchmark, at least the 9950X3D, and even bigger benefits for X3D in other use cases than code compiling, 9950X3D is only a few percent better than 9950X for that.

[wegfawefgawefg]

how about power consumption, cache size, etc?

what i know is that in my particle simulations and multithreaded compilation tasks it is easily 2-10x faster. where before i could choke on 1 million particles, now im choking on 30mill.

i specifically bought this processor for its rust compilation times.

[noosphr]

>what i know is that in my particle simulations and multithreaded compilation tasks it is easily 2-10x faster.

If Moore's law still worked like it did between 1975 and 2005 you'd be getting a 16x performance boost in _single_ threaded applications.

[genewitch]

The fastest single threaded, non overclocked, commercially available cpu was the Intel i3 7350k.

I haven't seen anything bench faster than a system set up for that machine on a business board.

I think now with advancements such as AVX (the newer versions that chips like atoms don't have) the 7350k can be viewed as a power hungry pig.

I'm not sure though. The systems I purpose built using those CPUs still work great, no real significant performance issues even with current software and web. And they can use a single battery UPS.

I still have a 16 core 5950x, that is 10% faster than the 40 core xeon box it replaced, at 1/4 the wattage at the wall as my main machine.

[KaoruAoiShiho]

It's much higher than 1.5x-2x, both scale up and scale out. Moving onto fp4 alone will offer a huge speedup.