[omeysalvi]

Most software innovations seem to be made to make the developer's life easier instead of creating more performant software. I'm not complaining but user focused software innovations like LLMs do seem to come very few and far between compared to hardware.

[wahahah]

I wouldn't consider LLMs a software innovation.

[karlshea]

I feel like if you're gonna drop a take like that you should have at least one more sentence of justification to follow it up with.

[Joeri]

I’m not sure I agree, but I’ll do my best to argue the position.

The neural net software algorithms have been around for decades. What made LLM’s feasible are the hardware advances to achieve unprecedented scale, just barely providing the ability (at great cost) to train today’s LLM models. Transformer architecture might be called a software innovation, but RWKV Raven gets similar performance to transformers and is built on decades-old RNN technology. So it is the hardware that was far more instrumental than the software in achieving LLM’s.

Counter to that argument: had google not done neural net research for google translate and proved the transformer approach scaled and performed well in their “attention is all you need” paper, people wouldn’t have spent the money to train foundation LLM models and we would not be having this discussion, so the software really mattered more than the hardware.

In reality I think it’s a little bit of both.

[NicoJuicy]

That's not necessarily true. The algorithm ( transformers) was done by Google in 2017.

Even while they missed the opportunity, Gpt-1 was released in 2018 by OpenAI.

Then they incrementally added parameters in the next versions.

[Shorel]

Great take. It is really a mix of several factors, each one leveraging the other, and your arguments are great.

[karlshea]

I don’t think I totally agree either but your argument for the position definitely has some merit.

[hkt]

I kind of admire the gumption, personally. Like writing them off as fancy markov bots.