[gwern]

It's quite amusing. The standard statistical theory does not work at all in estimating data vs model size, and the bounds are all vacuously large. It's a very active area of research, understanding why models act so simple when overparameterized and coming up with real measures of model complexity. Lots to read there if you are interested in such things.

[andrewprock]

That just means that the parameters are not independent.

[gwern]

But you can fit randomly-generated labels!

[andrewprock]

That's not in any way surprising. When you have more parameters than data, this is trivial.