[estevaoam]

Why most people are blind to the most important point of those models: the progress is moving at astonishing pace.

IMO it is a useless discussion to debate about the current SOTA or economics. In two years we will certainly have some breakthrough as we have been seen for the past years. And things are speeding up.

edit: typo.

[layer8]

> And things are speeding up.

That’s not how technical progress works. It’s not continuous. The past doesn’t predict the future. Progress may stall at any time.

[estevaoam]

Sorry, I didn’t want to sound too sarcastic and negative: I don’t disagree with you. But looking from a broader timescale it is clear that in general we’re speeding up our technological progress in a logarithmic scale.

I have been following closely the NLP field for over a decade and the progress is speeding up. Can it stall? Sure. But everything is pointing out that it won’t.

[smoldesu]

I kinda disagree. We've made some leaps and bounds, but our actual "progress" has mostly been defined by orgs like OpenAI throwing money at the problem. It's only technological progress in the sense that grinding stone bricks is "technological" or "progress".

The difference between ChatGPT and Talk to Transformer is frankly not that large, at least when treated as a black-box. 90% of the people freaking out over ChatGPT on Twitter would have also freaked out over the original GPT, had they known it existed. The extra nuance that we're adding on feels like stalling, and while some of the optimizations have been cool (gpt-neo-2.7b, Stable Diffusion) it feels like we're hitting the top of our progress curve.

[estevaoam]

Surely we can say that about our last 10s years of our human existence.

[arcturus17]

There are extremely interesting technical and economical critiques in the article. They may be right or wrong but at least they’re intellectually rigorous - much more than the average GPT commentary.

Your comment, in a nutshell and if I’m reading it right, is that it’s not worth to engage with any of these ideas because progress - whatever that means, however that’s measured - is fast.

[pixl97]

Unfortunately the measurement if something was worth engaging in is only really able to determine its value in hindsight.

If it were 1890 we'd be talking about how our cities will soon be buried in animal dung which will lead to the collapse of mankind. The people debating that could not have reasonably foreseen in 100 years that CO2 would be the greater risk, and 100 years from now the greater risk will be something most of us have not imagined.

Are the issues you point out worth talking about? Of course, but are they worth the amount of time and effort that we will debate them? Get back to me in 5 years and we'll see.

[Jensson]

> And things are speeding up.

Gpt-3 came 3 years after transformers were invented. Now we are close to 3 years after Gpt-3, did anything nearly as big happen during that time? Things aren't speeding up at all.

[fock]

> we will certainly have some breakthrough as we have been seen for the past years. And things are speeding up.

In the past years we have seen a tremendous scaling of a) workforce, b) hardware, c) data. For me a breakthrough would be on data/energy efficiency? What areas are promising there?