[itkovian_]

"That is a phrase that I coined in a 2022 essay called “Deep Learning is Hitting a Wall,” which was about why scaling wouldn’t get us to AGI. And when I coined it, everybody dismissed me and said, “No, we’re not reaching diminishing returns. We have these scaling laws. We’ll just get more data.”

How can anyone think he's arguing in good faith at this point. That essay was published after gpt3 prior to gpt 4 - and he's claiming it was correct!

[sytelus]

I lost faith in Marcus after just a few interactions. He is indeed what one would refer to as "crackpot" in academia. The most glaring thing was that he is technically extremely shallow and don't have a clue about most of the details. I also got impression that is enormously enamored with having attention and recognition at any cost. Depending on weather, he will change directions and views, basically just do anything it took to get that attention no matter how ridiculous he looks doing that.

While writing this, it occured to me that he would get even goose bumps at reading this comment because it, after all, I am giving him attention.

[baobabKoodaa]

> Depending on weather, he will change directions and views

My impression is the opposite: I would describe Gary Marcus as having all his opinions perfectly aligned to a singular viewpoint at all times regardless of weather (or evidence).

[sytelus]

At different points he has claimed (1) current models are not intelligent at all and can never be intelligent like us, (2) also we need to ban current models because they will outsmart us and take over the world.

Depending on how he can get interview or get seat at the table, he may chose exact opposite of positions.

[baobabKoodaa]

Can you show an example of 2)? I've missed that.

[nonrandomstring]

I thought "Attention Is All You Need"

[petters]

I don't think he is always arguing in good faith, unfortunately.

[abc-1]

So his timing was slightly off. I don’t know why people expected LLMs to improve exponentially. Your iPhone now doesn’t look much different than the one 10 years ago. GPT-3 or arguably GPT-4 was the first iPhone moment, everything else will be gradual improvements unless fundamental discoveries are found, but those happen seemingly randomly.

[anonylizard]

If one compares O3-mini's coding abilities to the original GPT-4. It is as large as GPT-3 to GPT-4 gap

GPT-3: Useful as autocomplete. Still error prone, but vastly better than any pre-AI autocomplete

GPT-4: Already capable of independently coding up simple functions based on natural language.

O3-mini: Can code in say top 5% of codeforces.

There's a 2 years gap between each of them.

More over, intelligence has a superexponential return, 90IQ->100IQ < 100IQ->110IQ in terms of returns.

[ben_w]

> More over, intelligence has a superexponential return, 90IQ->100IQ < 100IQ->110IQ in terms of returns

That's the second time I've seen the claim that linear increases in intelligence have "superexponential" results, first time was Altman's blog.

But I've not seen any justification for this.

(As you specifically say IQ, note that an IQ is defined as a mapping of standard deviations rather than a mapping of absolute skill, the normal mapping is 15 points being 1σ).

[rashidae]

AI is spreading across disciplines like science, math, software development, language, music, and health. You’re looking at it too narrowly. Human-computer symbiosis is accelerating at an unprecedented rate, far beyond the pace of something like the iPhone.

[PostOnce]

More like computer-human parasitism; it weakens the host.

It also only affects those with a "weak immune system" i.e. those whose bullshit filter doesn't function.

AI is here to stay for some tasks (segment anything, diffusion image generation for accelerating certain kinds of Photoshop), but LLMs are a dead end and AI Winter 2 is coming. They don't work for programming or law or medicine or mechanical engineering or even writing most emails because it's faster to just write the email yourself than to ask the AI to do it.

[mquander]

In what sense are the bleeding edge models incremental improvements over GPT-3 (read his examples of GPT-3 output and imagine any of the top models today producing them!), GPT-3.5, or GPT-4? Look at any benchmark or use it yourself. It's night and day.

Gary Marcus didn't make a lot of specific criticisms or concrete predictions in his essay [0], but some of his criticisms of GPT-3 were:

- "For all its fluency, GPT-3 can neither integrate information from basic web searches nor reason about the most basic everyday phenomena."

- "Researchers at DeepMind and elsewhere have been trying desperately to patch the toxic language and misinformation problems, but have thus far come up dry."

- "Deep learning on its own continues to struggle even in domains as orderly as arithmetic."

Are these not all dramatically improved, no matter how you measure them, in the past three years?

[0] https://nautil.us/deep-learning-is-hitting-a-wall-238440/

[abstractcontrol]

To me, the current LLMs aren't qualitatively different from the char RNNs that Karpathy showcased all the way back in 2015. They've gotten a lot more useful, but that is about it. Current LLMs will have as much to do with GAI as computer games have to do with NNs. Which is to say, games were necessary to develop GPUs which were then used to train NNs, and current LLMs are necessary to incentivize even more powerful hardware to come into existence, but there isn't much gratitude involved in that process.

[petters]

> To me, the current LLMs aren't qualitatively different from the char RNNs that Karpathy showcased all the way back in 2015.

It's very difficult to understand this statement. What meaning of "qualitatively" could possibly make it true?

[abstractcontrol]

The strengths and weaknesses of the algorithmic niche that artificial NNs are in hasn't changed a bit since a decade ago. They are still bad at anything I'd want to actually use them for that you'd imagine actual AI would be good at. The only thing that has changed is people's perception. LLMs found a market fit, but if you notice, compared to last decade where we had Deepmind and OpenAI competing at actual AI in games like Go and Starcraft, they've pretty much given up on that in favor on hyping text predictors. For anybody in the field, it should be an obvious bubble.

Underneath it all, there is some hope that an innovation might come about to keep the wave going, and indeed, a new branch of ML being discovered could revolutionize AI and actually be worthy of the hype that LLMs have now, but that has nothing to do with the LLM craze.

It's cool that we have them, and I also appreciate what Stable Diffusion has brought to the world, but in terms of how much LLMs influenced me, they only shorted the time it takes for me to read the documentation.

I don't think that machines cannot be more intelligent than humans. I don't think that the fact that they use linear algebra and mathematical functions makes the computers inferior to humans. I just think that the current algorithms suck. I want better algos so we can have actual AI instead of this trash.

[pj_mukh]

The difference between a doomsday conspiracy theorist and a physicist surmising the heat death of the universe is...just timing.

[suddenlybananas]

Well it's true that all of the most recent advances come from changes the architecture to do to inference scaling instead of model scaling. Scaling laws as people talked about in them in 2022 (that you take a base LLM and make it bigger) are dead.

[menaerus]

I think you want both. To scale the model, e.g. train it with more and more data, you also need to scale your inference step. Otherwise, it just takes too long and it's too costly, no?