[plastic-enjoyer]

Nobody's saying give up. I'm saying if your solution needs a trillion parameters and a power plant, and biology does it with a few billion neurons and a sandwich, that you're maybe on the wrong track. This is not an engineering gap.

[Zababa]

Considering progress in the rest of computing stuff (RAM, CPUs, storage) is kind of "linear"/exponential it sure looks like it's an engineering gap and we're on the right track. GPT 3 was 175B parameters and is today crushed by models that are 32B parameters, that's a lot of progress in 6 years.

[vidarh]

Typical storage densities have improved by several magnitudes per decade for decades. A mere 3 magnitudes of gap doesn't just seem like an engineering gap, but a relatively small one.

[pennomi]

I think the criticism is not “we need to have better hardware that will shrink over time”, it’s “our algorithms are hilariously inefficient, and nature shows that a better way must exist”.

Maybe there’s also a hardware component to it, but there’s very little point in trying to optimize the hardware to work with a poor algorithm. Once we discover an efficient way to train and infer, then it will be worth hyper-engineering the hardware.

[vidarh]

The point of the hardware comparison was not to argue that the hardware should be optimised, but to point out that it's ludicrous to point to the scale of the difference as evidence it's not an engineering gap when we have a history of overcoming far greater gaps in scale.

Yes, we need better architectures and algorithms. We can point to massive advances in software as well in many spaces, including in LLMs (e.g. compare early GPT versions with current smaller open models), but the hardware comparison came from further up-thread.