[islewis]

> "As long as your curve is sufficiently expressive all architectures will converge to the same performance in the large-data regime."

I haven't fully ingested the paper yet, but it looks like it's focused more on compute optimization than the size of the dataset:

> ... and (2) are fully parallelizable during training (175x faster for a sequence of length 512

Even if many types of architectures converge to the same loss over time, finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale.

[teruakohatu]

> Even if many types of architectures converge to the same loss over time, finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale.

This! Not just fastest but with the lowest resources in total.

Fully connected neural networks are universal functions. Technically we don’t need anything but a FNN, but memory requirements and speed would be abysmal far beyond the realm of practicality.

[actionfromafar]

Unless we could build chips in 3D?

[foota]

Not even then, a truly fully connected network would have super exponential runtime (it would take N^N time to evaluate)

[mvkel]

Wetware is the future.

[fennecfoxy]

Can't wait to see this defiantly spray painted across a torn up brick wall while computronium brained super intelligences slowly disassemble our planet to make paperclips.

[mvkel]

https://imgur.com/zeBkh2P

[ivan_gammel]

We need quantum computing there. I remember seeing a recent article about quantum processes in the brain. If that’s true, QC may be the missing part.

[tsimionescu]

This is just word salad.

There is no known quantum algorithm that can compute the result of a fully-connected neural network exponentially faster than classical computers can. QCs have a known exponential advantage over classical computers only for a very limited class of problems, mostly related to the Quantum Fourier Transform.

Animal brains have little to nothing in common to artifical neural networks. There is no reason whatsoever to think that there is any relation between the complexity class of brain functions and ANN inference.

And the hypothesized (and still wildly speculative) quantum behaviors happening in the animal brain are at the level of the behavior of individual neurons, not of the network connections between neurons. So even if there is some kind of quantum computation happening, it's happening in individual neurons, not at the network level, and that would only go to show even more that animal brains are profoundly different from ANNs.

[eru]

Compare and contrast https://www.smbc-comics.com/comic/the-talk-3

(Summary: quantum computing is unlikely to help.)

[bob1029]

We are already doing this.

[ComputerGuru]

Heat extraction.

[byearthithatius]

> finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale

Not to him, he runs the ARC challenge. He wants a new approach entirely. Something capable of few-shot learning out of distribution patterns .... somehow