[taeric]

I'm curious how representative the target function is? I get that it is common for you to want a model to learn the important pieces of an input, but a string of bits, and only caring about the first three, feels particularly contrived. Literally a truth table on relevant parameters of size 8? And trained with 4.8 million samples? Or am I misunderstanding something there? (I fully expect I'm misunderstanding something.)

[jaggirs]

I have observed this pattern before in computer vision tasks (train accuracy flatlining for a while before test acc starts to go up). The point of the simple tasks is to be able to interpret what could be going on behind the scenes when this happens.

[taeric]

No doubt. But I have also seen what people thought were generalized models failing on outlier, but valid, data. Quite often.

Put another way, it isn't just how simple this task seems to be in the number of terms that are important, but isn't it also a rather dense function?

Probably better question to ask is how sensitive are models that are looking at less dense functions to this? (Or more dense.). I'm not trying to disavow the ideas.

[visarga]

Maybe humans are also failing a lot in out of distribution settings. It might be inherent.

[taeric]

We have names for that. :D. Stereotypes being a large one. Racism being motivated interpretation on the same ideas. Right?

[lucubratory]

Yes, although there are less political examples. PTSD, the difficulty of learning higher dimensional mathematics in a way you can genuinely understand, substance abuse, mass killings.