|
> Assume for a second that a study of deep relevance to practitioners is replicated, and its conclusions accepted by the research community. It has large sample sizes, a beautiful research plan, a statistically sound system for controlling for various other explanatory factors As a multi-decade practitioner and manager of software engineering teams, I would certainly be interested in what the best of the empirical research has to say. I think it's important to always remain open to good new ideas wherever they may come from—strong opinions, loosely held, as they say. That said, I don't believe that statistically significant results can be found that will overturn my own instincts and judgement on any specific project to which I am dedicated. The reason for this is threefold: 1) the universe of software and goals we pursue with is astronomically large 2) competence in software engineering depends on the combination of personal aptitudes and mindsets combined with years of practice and 3) measuring outcomes in software engineering across diverse projects is all but impossible. In other words, you can't equate tools, you can't equate projects, and most of all you can't equate people. At the end of the day, success in software engineering comes from relentless focus on the specific goals at hand. One must be inherently curious and have a craftsperson's mentality about acquiring technical skill, but never become religious about methodology. This requires continuous first-principles thinking targeted at specifics. At the end of the day, two expert practitioners could propose unorthodox and diametrically opposed approaches to the same problem, and they would still dramatically outperform a lesser skilled journeyman who attempted to follow every best practice. Empirical studies and the scientific method in general work fantastically well for uncovering the rules and inner workings of the natural world, but software is the creation of logical systems purely by human minds which is an entirely different challenge—there's just not enough evidence to draw on. I suspect results will be at least a couple orders of magnitude softer than sociology, and that probably won't sit well with the type of personality attracted to software in the first place. |
This was pretty much the word-by-word argument against using statistical approaches to price insurance of shipments over the sea back in the 1700s. Yet we all know how insurance premiums are calculates today, and there's a reason for it.