[kosievdmerwe]

Sounds more like a formula that needs more terms.

Compare the classical physics formula for momentum, p=mv, vs relativistic formula, p=γmv. γ is almost 1 for most low velocities, it only starts jumping up to infinity when we get close to c.

The point being that the classical formula is pretty good in it's zone of low velocities, but as soon as you get too far out of the implicit term's "constraints" the formula breaks down and you need to add more to it to get it working for both low and high velocities. Which doesn't sound easy.

[nonbel]

Adding more terms sounds like more overfitting.

[jfengel]

It sounds like that only because scientists reformulate their models in terms that people are familiar with. Actual physicists don't work in terms of the gamma correction factor; they work in tensor fields that don't look anything like conventional arithmetic.

But they can pare all that down to something expressed in terms people are familiar with. And that has the bonus purpose of helping them understand why the familiar terms were familiar: the "correction factor" is small under circumstances we encounter, and only becomes large under circumstances we rarely do.

If that intrigues somebody enough to learn the actual physics, they'll encounter a completely different and more-encompassing formulation which looks not at all like overfitting. One that turns out to be more elegant, in fact, cramming more information into less notation. But it's information nobody needs until they're doing fairly advanced physics, so we're not going to be teaching it in elementary school any time soon.

[SiempreViernes]

Don't worry about it, I think you're talking to a NN stuck in a local optima around the term "overfit".

[nonbel]

I don't see how what you wrote could possibly address concerns with adding parameters to the model until it explains current data so perfectly that it cannot generalize to future or other data.

This is not a "theory" problem, it has to do with matching the theory to observations.