[moultano]

Unless you have an infinite regress on priors for your priors, and uncomputable Komolgorov penalties on the structure of your model, I think you need a test set. (This means you need a test set.)

[ced]

There's an interesting chapter in MacKay's book on Occam's razor. I'm not sure how I feel about it, but it's very thought-provoking.