[logicchains]

Most of the core idea of transformers was invented in the early 1990s, including what at the time were termed Fast Weight Programmers and are formally equivalent to linearised self-attention: https://people.idsia.ch/~juergen/fast-weight-programmer-1991... . Google just had the hardware to actually run and experiment with large transformers.

[7373737373]

It would be very funny if at some point someone stood up to Schmidhuber and told him they already created something he did, but yet another 30 years earlier.

[camjw]

At least you concede then that Google invested in the hardware.

[tsunamifury]

Jesus, the arrogance here. Are HNers this ridiculously out of Touch.

That is the investment and it was no small price. JUST is carrying a ton of weight here.