[jonex]

The problem with the observational approach is generally that it's really hard to decorrelate the results. In your proposed experiment you control only for the project itself. What if good developers generally prefer modern programming languages, so they chose Roslin. Does that imply that using Roslin is a better language for programmers that are not as good? Does it even show that they wouldn't have been even more productive using Java?

From the other direction, even if you get the value of controlling for the project itself, that might also add some bias. Could be that for a project with that setup waterfall actually works pretty well, but is it representative of projects overall? Are most software projects comparable to developing a simple Android app with a well defined specification up front?

I do agree that it would be good to do this kind of experiments where multiple teams get tasked with building similar systems to figure out what works. But I don't think it makes sense to actively avoid controlling for variables. That would make the results very hard to interpret and much less usable.