[Radim]

Re. your eloquent bracket: the "true number of bits", as in "the most compact description of any given state of a system", is in general uncomputable (see Kolmogorov complexity), for any sufficiently powerful language of description.

If you require notions such as "the true minimum number of bits" to be practical, you have to put additional restrictions on the language by which you describe the system -- such as your probability model. The representation does matter.