[11thEarlOfMar]

In the age of Internet, I believe there is a way to conduct experiments that would yield an answer based on data. Spitballing here, but how about a kind of contest for 'points' where a statistically significant number of devs volunteer to participate. They each provide a 'resume' (GitHub, LinkedIn, CV,...) into the system. The programming task is presented and the devs self assemble into teams, and endeavor to complete the task.

As an example, let's say 100 devs jump in. The task is to create a simple Android app, with a requirements statement provided, with server back end, launch it into the app store, support it for some period with bug fixes and improvements, and then declare it '1.0 released' to wrap up the experiment.

What you'd wind up with is a variety of team sizes, a variety of team experience, a variety of development systems used, a variety of outcomes. But all building the same software.

The key would be that as many attributes of each team's efforts as possible would need to be recorded and entered as data to be studied in search of patterns.

Repeat this n times and I believe valuable insights could be gained.

Rather than trying to control for all the variables of team size, experience, method, you control for the end product being targeted and then look for insights into the variety of approaches that teams took.

[jonex]

The problem with the observational approach is generally that it's really hard to decorrelate the results. In your proposed experiment you control only for the project itself. What if good developers generally prefer modern programming languages, so they chose Roslin. Does that imply that using Roslin is a better language for programmers that are not as good? Does it even show that they wouldn't have been even more productive using Java?

From the other direction, even if you get the value of controlling for the project itself, that might also add some bias. Could be that for a project with that setup waterfall actually works pretty well, but is it representative of projects overall? Are most software projects comparable to developing a simple Android app with a well defined specification up front?

I do agree that it would be good to do this kind of experiments where multiple teams get tasked with building similar systems to figure out what works. But I don't think it makes sense to actively avoid controlling for variables. That would make the results very hard to interpret and much less usable.