[sb]

I usually recommend E.T. Jaynes' "Probability Theory: The Logic of Science", which is one of the most beautiful and comprehensive books on the subject.

To get the "logic of science" part, you also need to have (IMHO) some fairly decent grasp of combinatorics, for which I quite recently stumbled upon one of the best books in this field: "Notes on Introductory to Combinatorics." (I like the links to many of Polya's gems of "How to solve it.")

For many other references, a quick HN search for publicly available references will result in other endorsements, too (a preliminary version of the Jaynes' book used to be available, too)