[madhadron]

For probability, you need measure theory first. That's also when discrete and continuous methods unify, so it's an amazing edifice.

Statistics has four major branches: inference, exploratory data analysis, experimental design, and visualization.

Your question is probably mostly asking about inference. I like the first few chapters of Kiefer's book to get the context of decision theory. Then you're going to have to wander far and wide, but there have been some good recommendations. Since you're digging into Dedekind on the side, you might like to go back to some of the classics here as well: Wald's "Statistical Decision Functions" and Savage's "Foundations of Statistics."

For exploratory data analysis, Tukey's book "Exploratory Data Analysis" remains the place to start, if you can find a copy that doesn't cost a fortune. Casella wrote a book on experimental design that's solid. For visualizations Wilkinson's "Grammar of Graphics" and Tufte's books are the usual recommendations.

[mhneu]

No no on measure theory. See below on Feller, Bersekas, Gallager. I think it's best to learn discrete prob first and skip the difficult math until you decide to become a mathematician or really need it for e.g. finance work.

One other objection: yes, those are fields in statistics. But statistics as a field is MUCH broader than departments called statistics. A ton of stats is done in engineering, psychology, and economics (econometrics). In fact I'd say the major research in stats for the past few decades has been done outside of "statistics" as a field. But yes, the books by Tukey, Tufte, and Casella are good.

If the OP is asking about statistics and probability needed for machine learning, he wants to focus on engineering statistics, like: estimation, stochastic processes, and filtering and mathematical statistics. The Wald rec is good and complements Feller.

[madhadron]

The OP is reading Spivak and Dedekind and makes no mention of machine learning. He's interested in the deep mathematics.