[kettleballroll]

As a member of the research community: that's nonsense. Like already pointed out: academic groups (who by no means are dependent on big tech) would jump all over that. Mamba has been out long enough that you'd already see tons of papers at arxiv showing mamba dominating transformers in all sorts of applications. But that's not happening, despite the ton of hype. That doesn't mean that mamba is nonsense. Just that it isn't the immediate transformer killer. It remains to be seen if something comes from it, eventually.

[godelski]

As a member of the research community: that's nonsense. Publishing is an extremely noisy process in ML and is getting increasingly difficult for smaller non big tech collaborating labs. Reviewers' go to are: more datasets, scale, not novel. The easiest way to approach this is to work off of pretrained models. This is probably more obvious in the NLP world.

I agree that Mamba doesn't solve everything and it still needs work. But I disagree with the logic that there isn't an issue of railroading.

[p1esk]

What’s the main difference between an ape’s brain and a human brain? Scale. So that’s the train we’re riding at the moment. No roadblocks yet, aside from cost.

[godelski]

> What’s the main difference between an ape’s brain and a human brain? Scale.

This is incredibly naive with absolutely no scientific basis. There is no evidence that this is in scale of data nor scale of architecture.

There are a number of animals with larger brains in terms of both mass and total number of neurons. An African Elephant has roughly 3x the number of neurons humans have. Dolphins beat humans in total surface area. Neanderthals are estimated to have had larger brains too! It isn't mass, neurons, neuron density, surface area. We aren't just scaled up chimps.

[p1esk]

Other animals with larger brains might have other bottlenecks preventing them from reaching full potential of their intelligence. Neanderthals might have been smarter than us, but went extinct for reasons not related to intelligence.

But my point stands - our brains have evolved directly from apes brains and the main difference between them and us is brain size.

[godelski]

> Other animals with larger brains might have other bottlenecks

>>> What’s the main difference between an ape’s brain and a human brain? Scale.

Your argument is inconsistent. Very clearly everything isn't scale or we'd use other things besides transformers. Different architectures scale in different ways and everything has different inductive biases. No one doubts scale is important, but there's a lot more.

[p1esk]

Scale is all we need for transformers (so far). It might also be all we need for ape brains. It’s not all we need for whatever elephant or dolphin brains evolved from.

When this stops being the case for transformers, we will need something else. I’m just pointing out it’s not the case yet.

[frozenseven]

Scary how someone can be so confident in their wrong information.

> An African Elephant has roughly 3x the number of neurons humans have.

An African elephant's brain is not a scaled up chimp brain in any way. African elephants have less cortical neurons than a chimp, and roughly a third of the amount that humans have.

> Dolphins beat humans in total surface area.

Animals even less related to humans and chimps, with no prehensile appendages, living in an environment where building stuff is exceedingly difficult. And of course their brains are obviously different from any great ape.

> Neanderthals are estimated to have had larger brains too!

And were just as smart as us and also had a scaled up chimp brain.

[godelski]

animal :: cortical neurons (b) :: total neurons (b)

Human :: 16 :: 86

Gorilla :: 9.1 :: 33

Chimp :: 6 :: 22

African Elephant :: 5.6 :: 251

Chimps are generally considered more intelligent than gorillas.

Bottlenose Dolphins have 11-15b cortical neurons while humans are in the range 14-18 (range is measurement uncertainty). It's also worth noting these dolphins have a larger brain mass (1.6 kg) and larger cortical surface (3700 cm2) than humans (1.3 kg and 2400 cm2, respectively).

> with no prehensile appendages, living in...

So more than scale. Glad we agree. Seems you also agree that architecture matters too.

[frozenseven]

> Chimps are generally considered more intelligent than gorillas.

And chimps are genetically more similar to humans than gorillas. A chimp brain is more similar to a human brain than a gorilla brain.

> So more than scale. Glad we agree.

We absolutely do not agree. Notice how nobody suggested that a human brain is a scaled up version of an axolotl's brain? Yeah, that didn't happen. I do wonder why?

[algo_trader]

> Just that it isn't the immediate transformer killer.

What is the best/stable-ish linear alternative for transformer right now? Especially for text generation and summarization.

We have domain specific ways of over sampling and search, so we much prefer less expensive models.