[pennomi]

I think the criticism is not “we need to have better hardware that will shrink over time”, it’s “our algorithms are hilariously inefficient, and nature shows that a better way must exist”.

Maybe there’s also a hardware component to it, but there’s very little point in trying to optimize the hardware to work with a poor algorithm. Once we discover an efficient way to train and infer, then it will be worth hyper-engineering the hardware.

[vidarh]

The point of the hardware comparison was not to argue that the hardware should be optimised, but to point out that it's ludicrous to point to the scale of the difference as evidence it's not an engineering gap when we have a history of overcoming far greater gaps in scale.

Yes, we need better architectures and algorithms. We can point to massive advances in software as well in many spaces, including in LLMs (e.g. compare early GPT versions with current smaller open models), but the hardware comparison came from further up-thread.