[est31]

Right but 2000 to 2010 didn't have similar progress, and especially 2010 to 2020 didn't. Sure, things have gotten better but not as much as the 1990 to 2000 leaps.

And yes, laptop specs haven't changed much and this is partially because the need for spec changes wasn't present, but also during the last 20 years there has been tremendous pressure for efficiency in datacenters.

Despite that, dennard scaling is dead since 20 years. There are physical limits. Already now, the wear effect of electrons jumping is present, and it will only get worse as things scale towards smaller sizes.

There are some benefits to be had, e.g. one can etch models into chips directly so you can pack them more closely, and run more inference on Tensor like chips, but that gives you maybe one order of magnitude improvement in total, at most. Also, of course nobody does that when each 2-6 months a new model comes out.

[marcus_holmes]

The thing that we did in 1990-2000 was adopt new standards as the old ones became blocks on progress.

I had a friend working in optical computing back in the late 80's that would wax lyrical about how optical computing was vastly superior to silicon back then. But it never took over because silicon worked well enough.

If we've hit the limits of silicon then there are other options. We would need to reinvent huge chunks of our tech stack, and that is incredibly expensive, but if the demand is there, we'll do it. The demand has never been there.

[saghm]

The original claim from the parent comment was running a Fable-level comment within a decade. Even if you're right about whether it's possible that another model could support that level physically, do you really think that we'll figure it out and ramp up the infrastructure to profitably sell on come consumer hardware anywhere close to that soon?

[marcus_holmes]

Well, we did do similar stuff back then. All it takes is money ;)

[saghm]

You're talking about going from a single gigabyte to 16 terabytes in a low-power consumer form factor, which is 1600x. A generous estimate of the factor of the RAM sizes for low-power consumer devices between 1990 and 2000 would still be a few orders of magnitude short, if I'm doing my math right.

Examples of what exactly you're claiming is precedent for this would be helpful.

[marcus_holmes]

This went down a rabbit-hole, which was fascinating, so thanks for the push :)

Not sure where the 1Gb number comes from? A standard laptop now is ~16Gb of RAM, so 1000x (and 1Gb -> 16Tb would be 16000x not 1600x). We went from Kb to Mb and then Mb to Gb of memory roughly every ten years from ~1990 -> ~2010. Each of those jumps is 1000x

Talking this over with claude, though, it pointed out that the need in dealing with LLMs is bandwidth and read-only storage, since the weights aren't dynamic. So we're not necessarily looking at 1Tb of RAM, we could be looking at 256Gb of faster RAM, and multi-TB of (much cheaper) flash storage, with extensive caching built in at OS level. This is all technically do-able with current tech, so it'll be interesting to see if it happens.