[YeGoblynQueenne]

This is yet another absurd statement in a long litany of absurd statements about this subject that I see constantly on HN.

And it is asburd because there is no such thing as "pre-training" is nature, much less anything done by evolution. Pre-training is very specifically a technique used on artificial neural nets, and then only specific kinds thereof.

Even in a metaphorical sense, an analogy cannot be drawn, between training an artificial neural net on many billions of instances of text, created by humans, carefully tokenised, or images, on the one hand; and, on the other hand, the evolutionary forces that operate in the real world, where there are no boundaries between the different sensory stimuli that animal nervous systems must learn to distinguish and manipulate.

What you're proposing is nothing but a "just-so" story. "How the squirrel got his brains and how the cockroach lost hers". By E. V. Olution.

[hackinthebochs]

>created by humans, carefully tokenised, or images, on the one hand; and, on the other hand, the evolutionary forces that operate in the real world, where there are no boundaries between the different sensory stimuli that animal nervous systems must learn to distinguish and manipulate.

You're making a very likely irrelevant distinction for the sake of undermining the parent's argument. If the point is wrong, argue that its wrong. But its certainly not wrong by definition, not in any substantive sense at least.

Yes, curation may be important such that it undermines the analogy between pre-training of a NN and an organism's evolutionary history. But you haven't argued the point.

[TeMPOraL]

> the evolutionary forces that operate in the real world, where there are no boundaries between the different sensory stimuli that animal nervous systems must learn to distinguish and manipulate.

Yup, and evolution spent millions of years to properly figure that out.

And "no boundaries between the different sensory stimuli" doesn't even look like a hard problem. If anything, the tricky bit was to identify what's useful to sense, and to make the hardware for it - which is something evolution has been figuring out since the first replicator, and it identified half of the useful senses even before multicellular life became a thing. At this point, the boundaries are already established - you have different types of sensors producing correlated, but still quite distinct signals. It's not that hard to keep those feeds separated.