[dvt]

I beg to differ. Transformers are purely an optimization. It’s not exactly right to call everything “compute scaling” but we are still, at the end of the day, fitting polynomials.

And frankly, that’s probably not what our brains are doing.

[bjornsing]

It is a bit weird though… I mean, you could just as well say that the last breakthrough in all of computing was the transistor (Shockley 1946) – all we’ve been doing ever since is just “scaling” and combining them in new ways.

[namaria]

AI has been a 'moving aim' as opposed to moving target. It's a label slapped on whatever slice of software engineering seeming most magic-like at any given decade.

Early computers in late 40s were called 'electronic brains' by the media...

[f6v]

It’s difficult to comprehend impact of something that’s widely adopted. Like, batch normalization alone was probably mind blowing when it came out. Yet it seems so simple and self-explanatory now.

[imtringued]

Or residual connections. It is just addition.

[mitthrowaway2]

> And frankly, that’s probably not what our brains are doing.

I think that it is! It's much more likely to me that our brains are doing something big and simple than small and complicated. That's the way that nature tends to work. Fitting low-order million-dimensional polynomials would meet that description.

[southernplaces7]

Our brain is however the very definition of small and complicated. It's the most complex known organic object known to humans and for all that any given normal brain handles in a given day, it uses just 0.3 kilowatt hours (kWh) to do it. No computer we have comes close to handling what a brain handles with that power consumption.

ChatGPT by contrast, consumes roughly a gigawatt hour per day serving its users. Yes, to do this it's handling a colossal amount of queries, but that's all it's doing. Your brain handles everything in your body and consciousness, in ways we don't even fully understand, while also letting you think and communicate and reason as a conscious being with self direction.

Moreover, there is evidence that at least part of our brain's functions may be exactly as the other reply here mentions, weird, subatomic and deeply complex in ways that are difficult to get a clear grip on.

[dvt]

> That's the way that nature tends to work

From the double slit experiment, to particle-wave duality, to the particle zoo of the 70s, to quantum chromodynamics, to asymptotic freedom, to more exotic theories like string theory, etc. tells us the complete opposite. Every major discovery in physics in the past 150 years seems to disagree. Things are extremely weird and complicated when we get extremely tiny. Why would our brains be different?

[mitthrowaway2]

If we lived at the quantum scale, then classical physics would be the weird one. Quantum chromodynamics is only confusing for two reasons: it differs from our everyday experience so we don't have an intuition for it, and because it has a large number of mutually-interacting (but basic) components.

Richard Feynman put it very well:

"The world is strange, the whole universe is very strange, but see when you look at the details then you find out that the rules are very simple, of the game, the mechanical rules by which you can figure out exactly what's going to happen when the situation is simple. It's again this chess game; if you're in just the corner with only a few pieces involved, you can work out exactly what's going to happen. And you can always do that when there's only a few pieces. And so you know you understand it. And yet, in the real game there's so many pieces you can't figure out what's going to happen.

"There's such a lot in the world, there's so much distance between the fundamental rules and the final phenomena that it's almost unbelievable that the final variety of phenomena can come from such a steady operation of such simple rules... But it is not complicated, it's just a lot of it."