[ghm2199]

Data science is always a very overloaded term ever since it took off way back in the 2010s. One of, if not the most, durable definition of this that likely can also be the most valuable because it’ll probably land your jobs(even today) is being able to make decisions from looking at data that have an team wide(good IC like job security) scope at the least and company wide scope(very rich).

Building that intuition is incredibly difficult. It can be learned if one likes to solve and think about problems that way. Like for example you can get quite far with knowing how to use linear regression(for example coefficients of linear regression can be determined using a deterministic algorithm using linear algebra yet knowing the assumptions of linear expected value and constant or variance is more useful as is the knowledge of what probability model to use to define the random variable(hmm are these Bernoulli events or poison)).

How to do sampling(like using reservoir sampling when you have an infinite sample count e.g in a long running crowd sourced survey to not over or under sample buckets for calibration).

Or just rule of thumbs like how # of samples needed for moving decimal point on significance varies roughly as inverse of sqrt of N and probably much more in case of interacting factors.

I would like a book on that :)

[jubilanti]

The discipline you are looking for is called Statistics

[lopsotronic]

I have heard it confidently stated, repeatedly, by extremely technical coder types, that "once we have enough data, we don't need Statistics"

It's a pretty funny sentiment.

It's also, unfortunately, reflective of how even talented individuals pass through advanced accreditation programs without locking blades with - or at least entering a general understanding of - fundamentals. Old timey British boarding schools would have called it "Logic" back in the day.

I am reminded once more of the differences in MSFS and Xplane: one uses a statistical table based model based on existing aircraft, and the other uses laminar flow analysis, fluid dynamics, and physics. One of them tells you something about unknown aircraft, and the other most definitely does not.

[ghm2199]

I’ve had the good fortune of taking two courses at Columbia in the Social Sciences department(Andrew Gelmam and Ben Goodrich teach there). I think they probably are right up there if not the best at trying to teach students how to work practically with Statistics(specifically Bayesian statistics). Though they have always lamented that most schools do a poor job of teaching it such that kids can apply it.