[rrmm]

You also have to know you want to solve a problem.

Once you get to the point where you have any hypothesis whatsoever, no matter how weak, a systematic approach (saving examples as test cases) helps to avoid confirmation bias and makes testing further hypothesis less costly.

Another hard one is when there is a simple, probabilistic rule. You usually end up with an over-complicated rule to cover all your data instead of the true rule. (Of course that gets down to what is at the basis of the probability: are you satisfied with a probability?)

[skybrian]

Probabilistic rules themselves tend to require much more data, which can be expensive.

In computing we try to write deterministic tests that either pass or fail, which means you can run them once after a change and know what the state is. Even if you just suspect flakiness you may have to run the test hundreds of times to be confident that the probability of failure is sufficiently low.