[sgt101]

There are a lot of subtle points to make here. There is a tendency to throw data at models that don't capture parts of a distribution, and it is definitely true that many of the tail events in a challenging domain will not occur again no matter how long we observe the domain. Successful machine learning systems are able to predict these outcomes without having seen the data previously because they have captured the theory that creates them. Unfortunately it is very difficult to determine when a model is capturing the domain theory and when it is just modelling a distribution - often the only way is to "know" that it's a bit fishy. In many domains this difference doesn't matter, vision in animals seems to work in this way - it's all approximations and sameasis, and we and machines get tricked by optical illusions and so on. Other domains (many in physics) are modelled by observing data and inferring a higher level theory. Early days physics didn't work this way - Chomsky is right, but the method of Galileo is not the only method. Modern scientists do organise data and do look for exceptions and regularities which then drives the search for explanatory systems with predictive power.