[zmmmmm]

I obviously have not looked at all (or really much) of this research, but I have always felt that software development is so context dependent that drawing generalised conclusions is just an impossible ask in the first place. Even if you did, there would be so many exceptions that in practice it will come down to "use your experience to assess the context and then decide".

The example of TDD : I've done significant pieces of work both with and without TDD. In some scenarios its a huge impediment; the actual complexity of the internal software is fairly low and the but the testing complexity is high (many complex stateful dependencies that are hard to control). In that case, I spent 80% of my time writing the tests and far more bugs surfaced in the tests than the code itself.

Then in other scenarios there's high internal complexity, low external dependencies / complexity and it's pretty much a no-brainer, TDD is almost the only tractable way to write the code let alone an improvement.

Then it's very personal as well. One person will work well with TDD and another will struggle. Dumb things like, is your personal preference in development environment conducive to rapidly running and iterating on tests are probably going to dominate.

End result is, I think these studies just can't possibly control all the variables and this is why they either end up in invalid conclusions, too specialised conclusions, or, as Jimmy says, the more rigorous the study the less significant the result are.

[rTX5CMRXIfFG]

It seems to me that you misunderstand the purpose of statistical generalizations in the first place? The purpose of looking at multiple cases is to derive common patterns regardless of the differences in detail among individual cases. Which is why, yes, you still need to apply the inferences into a specific context if you need to make a decision, but to say that nothing can be gained from a comparative big-picture approach is a malicious withdrawal from critical thinking.