[rossdavidh]

In real-world software development, managing complexity is often (usually) the core of the challenge. A simplified example, is leaving out the very thing that is the obstacle to most good software development. In fact, it is sometimes the case that doing something that helps with managing complexity, will impair performance as measured in some other way. For example, it may slow execution speed by some amount, but allow the software to be broken into smaller pieces each of which is more compehensible. Managing this tradeoff is the key to much software development. If you test with "toy experiment design", you may be throwing out the very thing that is most important to study.

[mmooss]

Great point. I think that also emphasizes the necessity of the D in R&D: The research has to be adapted to the real world to be useful, for example to organizational frameworks and processes that manage complexity as you say.

Most software organizations I know don't have anything like the time to do D (to distinguish it from software development), except in a few clear high-ROI cases. Big software companies like Microsoft and Google have research divisions; I wonder how much they devote to D as opposed to R, and how much of that is released publicly.