[westurner]

Is the limit: A) sensor resolution, B) NN architecture and/or algorithm, C) training sample size, D) training data (labeling, segmentation) quality, or E) it doesn't sufficiently predict the variance with low enough error?

New NN models are able to do more with the exact same sensor data.

[_hl_]

You cannot conjure information out of thin air. Even with infinite data and a hypothetical wormhole CPU that runs everything in O(1) and solves the halting problem, you still couldn't do this. So to answer your question, the reason is effectively (A). Sensor resolution might be the wrong term but it's the general idea.

[westurner]

How much information content is there in DNA (and RNA,)? How do creatures know or learn what not to eat given limited available sensor data?

[dekhn]

How much information content is there in DNA? 2 bits per base, before compression. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3220916/

How do creatures know what to eat? Evolution solved that for most creatures, so their sensors don't have to work as hard at runtime. And in other cases, some number of members of a population of creatures will die before the population learns the food is poisonous. Our sensors, and the information processing systems that manage their outputs, are remarkably efficient data processing engines that do the equivalent of approximating and predicting, often well beyond what the most advanced deep learning systems are capable of doing now.

[westurner]

So, sensor resolution is higher, there are multiple fields being integrated, in a massively-parallel spreading-activation Biological Neural Network, and that's how blank-slate creatures just know?

Is there enough information content - per the Shannon entropy definition or otherwise - in DNA and/or RNA to code for the survival-selected traits that

I'm not sure that the (Shannon entropy, MIC, Kolmogorov,) information content of the samples is the limit of any given network trained therefrom? Is there anything to be gained from upsampling and adding e.g. gaussian blur (noise)? Maybe it's feature engineering, maybe it's expert methods bias, maybe it's just sensor fusion; that's the magic noise.

[westurner]

Perhaps this is moving the goalposts a bit, but e.g. depixelation does appear to defy such a presumed limit due to apparent information content? Perhaps it is that the network reading the sensor carries additional information associated with the lower-resolution or additional-fields' sensor data?

https://github.com/krantirk/Self-Supervised-photo :

> Given a low-resolution input image, PULSE searches the outputs of a generative model (here, StyleGAN) for high-resolution images that are perceptually realistic and downscale correctly.

Maybe no amount of feature engineering can actually add information?

[_hl_]

Because they receive additional information from the environment through highly sensitive sensors producing massive amounts of information. Whereas the information you get from a cheap sensor effectively discretizes to a few bits.

[dtech]

You can, but it's called making stuff up

[analog31]

Having designed sensor systems, I've lost more than a few hours of my life having to explain "why do we need that big expensive sensor when you can do everything with machine learning?"

The idea that a magic math technique can replace expensive sensors predates NN's by a few decades. Dozens of start-ups have gone bankrupt trying to do non-invasive blood glucose with portable sensors.

This is a very crude but at least conceptually useful rule of thumb: It's all of the above, but ultimately the analysis result is a mathematical function of an array of values produced by the sensor. Very few math functions do not have the property, that variation in the output increases with the level of variation in the input.