[anonylizard]

Investment is not the sole input into creating technology, time matters to. The tranfsormer paper probably didn't take much investment to create, but without it, any attempts at creating LLMs would have been futile no matter the billions poured in.

Moreover, technology is created as spillover effect from investments in other areas (That aren't speculative). CUDA is only possible because Nvidia was supported by gaming revenue for a decade. Without CUDA, there is no alexnet, and no deep learning boom will happen if we are still on CPUs or google's proprietary TPUs.

[tsunamifury]

I’m sorry … but what?! As someone who worked on early LLM tech at Google hundreds of millions were poured into it. Do you think that paper just spontaneously came from pure theory?

[logicchains]

Most of the core idea of transformers was invented in the early 1990s, including what at the time were termed Fast Weight Programmers and are formally equivalent to linearised self-attention: https://people.idsia.ch/~juergen/fast-weight-programmer-1991... . Google just had the hardware to actually run and experiment with large transformers.

[7373737373]

It would be very funny if at some point someone stood up to Schmidhuber and told him they already created something he did, but yet another 30 years earlier.

[camjw]

At least you concede then that Google invested in the hardware.

[tsunamifury]

Jesus, the arrogance here. Are HNers this ridiculously out of Touch.

That is the investment and it was no small price. JUST is carrying a ton of weight here.