This completely misses the concept that the technology doesn’t just create itself without investment. SOMEONE has to invest for that progress to be made. It’s why I always hated the application of this to tech.
> SOMEONE has to invest for that progress to be made.
History often has cases where the surrounding context/infrastructure/market/culture just wasn't suitable for an invention to take off--so it fails--and then somebody 5-100 years later tries something almost the same thing and it becomes a wild success and they get all the credit for it.
For example, look at the AT&T Picturephone from ~1964. [0] Even if someone from today time-traveled with the benefit of ~60 years of economic and technological hindsight... do you think a strategy of "just keep investing--somebody has to" would been enough to make the video-phone a widespread and profitable thing in any reasonable timeframe?
I'm pretty sure you'd go quite bankrupt first, just trying to stay solvent while making hundreds of other enabling-inventions and cost-savings in a wide range of fields while completely replacing global infrastructure.
But we did people continuously invested through the 80s then 90s then Cisco then Google and now zoom. Do you and the writer not know that video calling was a consistent investment for over 40+ years?!
By that logic, somebody was so desperate to get convenient warm food that they decided to "invest" for decades in side-projects like airplanes and bombs just to finally get a microwave-oven, obviously true purpose of everything that came before.
Just because something else in civilization might somehow help someday isn't enough to make it part of "the investment" which a company or individual is deciding whether to make or abandon for a different goal they are focused on.
Sorry but I feel like you are trying to be clever here but failing entirely because yes that is how it works. Continuous investment yields unexpected outcomes is like the core tenet.
Dude, the core tenet of this forum is making a good-faith [1] effort to really understand the fantastically complex issues at play here, and then to communicate about them honestly in a collaborative, rather than competitive, spirit.
Investment is not the sole input into creating technology, time matters to. The tranfsormer paper probably didn't take much investment to create, but without it, any attempts at creating LLMs would have been futile no matter the billions poured in.
Moreover, technology is created as spillover effect from investments in other areas (That aren't speculative). CUDA is only possible because Nvidia was supported by gaming revenue for a decade. Without CUDA, there is no alexnet, and no deep learning boom will happen if we are still on CPUs or google's proprietary TPUs.
I’m sorry … but what?! As someone who worked on early
LLM tech at Google hundreds of millions were poured into it. Do you think that paper just spontaneously came from pure theory?
Most of the core idea of transformers was invented in the early 1990s, including what at the time were termed Fast Weight Programmers and are formally equivalent to linearised self-attention: https://people.idsia.ch/~juergen/fast-weight-programmer-1991... . Google just had the hardware to actually run and experiment with large transformers.
It would be very funny if at some point someone stood up to Schmidhuber and told him they already created something he did, but yet another 30 years earlier.
Yeah information is different since once you find something out its often trivial to replicate and deploy it. The cost is coming up with the knowledge in the first place.
This is the problem with predatory game theory. Yes it’s “smarter” but it requires people to take the loss and get eaten for the “smarter” players to smugly claim victory. And if everyone acted like them nothing would work.
I hate it and I hate the arrogance of essays like this. They dive deeply into entirely missing the point.
History often has cases where the surrounding context/infrastructure/market/culture just wasn't suitable for an invention to take off--so it fails--and then somebody 5-100 years later tries something almost the same thing and it becomes a wild success and they get all the credit for it.
For example, look at the AT&T Picturephone from ~1964. [0] Even if someone from today time-traveled with the benefit of ~60 years of economic and technological hindsight... do you think a strategy of "just keep investing--somebody has to" would been enough to make the video-phone a widespread and profitable thing in any reasonable timeframe?
I'm pretty sure you'd go quite bankrupt first, just trying to stay solvent while making hundreds of other enabling-inventions and cost-savings in a wide range of fields while completely replacing global infrastructure.
[0] https://www.youtube.com/watch?v=BQMnlKMFD8M