[fwlr]

It does re-ignite an old interest I have in, I don’t know the right name for it, but I think of it as “seafloor” neural networks, where they aren’t equally deep in layers all across the network (as opposed to “swimming pool” nets which are a constant layer depth at every point. From my limited attempts at neural net inspection I recall seeing some nodes acting like passthroughs where it seemed to be able to do the operation in e.g. five steps and “didn’t need” the other three so those three layers had been trained to change the values they were given as little as possible. No idea if a network would be able to “push around” operations so the simpler ones end up in the “shallower” parts of the network or not, but gradient descent is pretty amazing sometimes.

[tysam_and]

Are you talking about residuals? They can be extremely helpful for a variety of reasons.