[fpiacenza]

Interesting, but this approach is still about scaling in size. There are other more promising approaches that actually allow to reduce size of the network and have better performance: Closed-form continuous-time neural networks [1], or also known as liquid neural network.

They have been proved better in navigation (cars and even drones), with as little as 15 neurons in the network. The smaller scale not only makes it highly efficient, but also allows to better understand what the network is doing due to lower complexity.

[1] https://www.nature.com/articles/s42256-022-00556-7

[fwlr]

It does re-ignite an old interest I have in, I don’t know the right name for it, but I think of it as “seafloor” neural networks, where they aren’t equally deep in layers all across the network (as opposed to “swimming pool” nets which are a constant layer depth at every point. From my limited attempts at neural net inspection I recall seeing some nodes acting like passthroughs where it seemed to be able to do the operation in e.g. five steps and “didn’t need” the other three so those three layers had been trained to change the values they were given as little as possible. No idea if a network would be able to “push around” operations so the simpler ones end up in the “shallower” parts of the network or not, but gradient descent is pretty amazing sometimes.

[tysam_and]

Are you talking about residuals? They can be extremely helpful for a variety of reasons.

[tysam_and]

I really love small neural networks. They have some nice properties that people overlook. The training speed record (warning, self promo) for CIFAR10 to 94% uses a very tiny neural network (<10 MB if just saved raw out to disk as a definition file). That's located at https://github.com/tysam-code/hlb-CIFAR10.

You could make that even smaller if you wanted to, though at least this network is already pushing maybe even a little further down the diminishing returns spectrum in some areas than I'd like, I suspect (bound by the powers of 2/multiples of 8/multiples of 64 required by GPUs :'(((( ).

I think a really fun future challenge (yes, I know, this is taking us back to the 90's-2000's in terms of challenge territory, but I believe it has much more practical use than a lot of other modern day benchmarks) would be to find the fastest-training network that infers at 94% in under 1 MB. I certainly believe it's possible, but with pareto laws the way they are, it would take a whole lot longer to train and might not be as fast on a GPU during inference as the main net (despite having fewer parameters). That might not be true, however.

There's a few NP-hard problems that actually exist in this space that not a lot of people talk about but I feel will be considered a core part of the theory of training neural networks at some point in the future. The size of the network is a very interesting tradeoff that opens up certain mathematically interesting properties on either end of the spectrum. Bigger is not always better, though it is simpler and simple oftentimes survives, especially (especially especially especially) when skilled technical workers in a very particular niche live in the shallow long tail of [insert correct distribution of skills rarity here].

One of the common threads (might be a "common", I'm not sure to be honest as I live in my own personal bubble of research interests and community and etc) is the dimensionality of the problem at hand. That plays into the scale of the network used to solve a problem. I remember some discussion being sparked a while back from some Uber research about the inherent dimensionality of a particular problem for some given particular neural network (though of course it's naturally linked to your inductive bias so please take that as you will). As you noted, some networks do quite well with very few neurons, 15 is a record however from what I've heard (and I'd love to see that -- I have a guess as to which particular method, or, at least, method family, it is... ;P I'm...casually interested in that arena of research).

In any case, as you can see I am quite interested and passionate about this topic and am happy to discuss it at length further.