[seanwilson]

> Try this thought experiment: Imagine two teams of programmers, working with identical requirements, schedules, and budgets, in the same environment, with the same language and development tools. One team uses waterfall/BDUF, the other uses agile techniques. It’s obvious this isn’t a good experiment: The individual skills and personalities of the team members, and how they communicate with each other, will have a much bigger effect than the methodology.

Another thought experiment: imagine getting two teams of programmers using the same methodologies and everything else and expecting the results to be the same. It's just not practical to perform studies like this because there are too many variables.

[gordaco]

That's because you are using an extremely small sample; of course the individual differences are going to matter. Now, get a random sample of 1000 teams using waterfall and 1000 teams using agile, and you'll get much more meaningful data. The larger the amount of samples, the more the individual differences are going to be smoothed.

Of course, not many people have the resources to do that kind of experiment.

[seanwilson]

> Of course, not many people have the resources to do that kind of experiment.

That's what I meant about it not being practical. Who would invest that amount of money? There's infinite variations of different methodologies as well.

I'm not sure what the solution is but it's tiring seeing bad studies used to promote certain approaches.

[coldtea]

>That's what I meant about it not being practical. Who would invest that amount of money?

Tons of organizations and governments could.

20 10 person teams * 5 methodologies = 1000 people, for a 1 month project = 1000 * $100.000 = 100M dollars.

In the grand scheme of things this is insignificant amount -- in a world where businesses spend $15 billion for buying Instragram. A military could do that kind of spending for buying a single airplane -- and such a research could potentially have a huge impact / savings on future soft-eng projects.

And it could be even subsidized or be tax-deductible. Or could easily drop to like $70K or $50K per month compensation.

[rtpg]

How about: You have two teams. Each team will work on two projects, A and B together.

The first team does A in "Agile" and B in "Waterfall"

The second one does A in "Waterfall" and B in "Agile"

I bet working like this would pull out at least some interesting stuff

[seanwilson]

It would still result in basically anecdotes. Maybe team A picked a library that made life difficult for themselves or misinterpreted one of the requirements? Small things completely throw the results off.

[BigJono]

Funny you mention that. The only "methodology" I feel I have enough anecdata to comfortably vouch for is YAGNI. In my experience, whichever team fucks themselves with bad library choices is going to lose this experiment 100% of the time, agile be damned.