[vervez]

Here's a good recent paper that looks at this problem and provides remedies in a Bayesian manner. https://arxiv.org/abs/2202.11678

[tesdinger]

Quote

|How do we compare between hypotheses that are entirely consistent with observations?

|... Occam's razor

The answer is, build an experiment that makes it obvious. According to Occam's Razor, the Sun was burning coal, before we found out, it doesnt.

[vervez]

Their paper is more for the case of "we can't gather more data, so what to do?" but your solution is in line with optimal experimental design and choosing a utility function to distinguish between models using as little data as possible.