[sgt101]

Well - that's true apart from co-incidence. You can have a very simple theory which says "x is directly caused by y" and there is a lot of good data, and a great fit. But it's kist a co-incidence and breaks down immediately.

Occam's razor is a rule of thumb and an aesthetic boon, but nothing more.

The real test is that you have a theory that is meaningful and has explanatory power. If it grants insight on the mechanisms that are driving the relationships or generating the data and these make sense - you are pretty golden.

Another one is that the theory makes unexpected predictions that you can then test. This is a real winner, and why complex physics is so well regarded.

[jwp729]

I think the information theoretic approach to modeling concerns actually implies such "simpler is better" approaches as Occam's Razor. At least that's my take on [http://arxiv.org/abs/cond-mat/9601030], which derives a quantitative form of it.

[sgt101]

I haven't read that paper, and the abstract makes my head spin! I'll have a look later, and try and figure out the argument. I agree with you that things like the I-measure are based on the idea that simpler is good, and it works well in practice - both in Machine Learning and in the real world - which is why humans tend to prefer it. But (the paper you cite aside) I don't know of a deep reason why simple is preferred by nature.

Also there is a deep cognitive bias here, perhaps we lack the machinery to understand the world as it really is!

[deciplex]

> Occam's razor is a rule of thumb and an aesthetic boon, but nothing more.

Occam's razor is a bit more than that. It isn't just that given a theory X and a theory Y = X + ε, both of which fit the facts, you should prefer X because it's "cleaner" or more aesthetically pleasing or whatever. You should prefer X because you can prove it is more likely to be true.

https://en.wikipedia.org/wiki/Solomonoff%27s_theory_of_induc...