[jdlshore]

I'm disappointed to see the 10x programmer meme hit a publication like the Economist. There's no scientific evidence of 10x differences in programmer productivity. To quote Laurent Bossavit [1], who's investigated the studies people like McConnell use:

"How strong is the support conferred to the 10x claim by the best-reputed list of references, for a reader persistent enough to follow the chain of citations back to primary sources?

"Based on our close reading of the “10x files”, we can now answer: quite weak.

"Not a single one of the references is to a replication, in the scientific sense of the term, of the original exploratory experiment.

"The empirical data is in general quite old, most if not all of it predating widespread use of the Internet - which we can safely expect to have wrought major changes in programming practice.

"None of the studies address the question of construct validity, that is, how meaningful it is to speak of an individual programmer’s productivity, and if it is meaningful, whether the experimental measurements line up adequately with that meaning."

Don't get me wrong--I've certainly seen 10x (and greater) variations in productivity. But they've been due to factors other than inherent individual capability. They're things like

* technical debt

* familiarity with the codebase and problem domain

* interruptions, distractions, multi-tasking and other office environment issues

* choice of language - but this is a distant fourth compared to other the other three

On top of all this, the Economist article is a thinly-veiled press release for tenXer, which is itself nothing more than an exercise in testerone-fueled egotism. tenXer measures activity, not productivity [2], and I expect the primary result of increasing your tenXer metrics will be less time thinking, more time hacking, and insufferable smugness.

[1] The Leprechauns of Software Engineering explores what science is and how we distinguish between fact and folklore in software engineering. It specifically explores the 10x claim, and determines that it's folklore. http://leanpub.com/leprechauns

[2] Productivity is defined as output over input, and the output of programming is software capability. No one's come up with a good measure of that yet. So any time someone claims they can prove increased productivity, ask them how they measure it. Chances are good you'll get a bogus value that measures activity instead, such as "lines of code." http://martinfowler.com/bliki/CannotMeasureProductivity.html

[btilly]

One of the better sources for these differences and their magnitude is Peopleware. Instead of citing vague research "somewhere", they did their own primary research. More specifically they set up coding wars where different programmers at different companies assigned to the same task independently.

They found 10x differences in time spent on the average coding war (which was typically a pretty small sample) with those finishing faster typically producing programs that worked better. So by any reasonable measure, at least 10x productivity.

There is your 10x, right? Wrong. They found that the best predictor of programmer performance was the performance of another programmer at the same company. Furthermore they managed to correlate a lot of that performance factor to specific factors, such as having a phone that turned off, a private room, adequate desk space, and so on. There was still something like an unexplained factor of 3 left over, but they didn't know whether it was environment variables they had not looked at (eg training), individual ability (which might be correlated across institutions), etc.

It is true that this research happened in the 80s, before the Internet was widespread. I would love to see it replicated again.

(Note, I'm aware of the existence of many other poorly controlled studies, see http://www.flownet.com/gat/papers/lisp-java.pdf for an example, but what is critical in the Peopleware study is that they acquired information both about productivity and about factors that might cause productivity differences.)

[pradocchia]

In Peopleware, they also observed that some programmers did not themselves appear to produce much, but their presence on a team made others more productive. They were not managers, they were not leads, they just helped the team work better together, in some fashion, "managing" out or across rather than up or down.

I will jump the shark and speculate that these gel-members tend to be female more often than male, and the gelling-like effect comes from how their presence and patterns of behavior influence the group dynamic. I will further speculate that the dearth of women in development incurs a hidden cost to productivity, since statistically, fewer teams will have women, and consequently, the incidence of gel-members will be lower, and teams more fragmented.

I think there is a business opportunity here somewhere, or maybe it has already been realized in places, but the players have not been eager to trumpet their results.

[MartinCron]

The thing about these studies, which I do appreciate, is that they are measuring a programmer's ability to work on something:

1) Small.

2) By themselves.

3) Well understood and documented.

4) Doesn't have any threat of changing out from under them.

In the real world, the bottleneck usually isn't writing code, it's writing code that does the right thing and making the right people happy in the right way.

In some ways, trying to apply the lessons from Peopleware in the real world is like using your knowledge of unicorn anatomy to predict the Kentucky Derby.

[barrkel]

I've seen developers who create, rather than reduce technical debt; who take longer to get familiar with codebases; have a hard time focusing in the presence of distractions; and who seem to have difficulty ramping up solution domain knowledge (i.e. programming / software engineering techniques). And all of these things seem to be down to individual ability or motivation, i.e. something about the person, rather than external factors.

Most developers suck at programming (and that's actually OK). 90% of the code they write is copy and pasted from elsewhere in the codebase or from the web, and then tweaked until it does something approximately correct. They usually need help when solving a new pattern of problem. What they do often have is domain knowledge not related to programming, and produce just about acceptable results out of the machine in a labour-intensive way that doesn't really scale over time or problem size, but they muddle through with determination.

I don't know about a 10x productivity differences though. I'm not really arguing in favour of that hypothesis. Rather, if anything, I've seen step differences in potential; that is, there are developers that can do things other developers can never do, not in 10x the time, not in 1000x the time.

[zmj]

> I've seen developers who create, rather than reduce technical debt; who take longer to get familiar with codebases; have a hard time focusing in the presence of distractions; and who seem to have difficulty ramping up solution domain knowledge (i.e. programming / software engineering techniques). And all of these things seem to be down to individual ability or motivation, i.e. something about the person, rather than external factors.

I don't entirely disagree with you, but watch out for the Fundamental Attribution Error: http://en.wikipedia.org/wiki/Fundamental_attribution_error

[barrkel]

My worthless opinion is based on about 20 years of newsgroup / forum membership, but the few years I spent working in companies whose software product was a cost centre for its customers was particularly enlightening. We (the guys working on framework / architecture stuff) tried to grow some of the people from the domain-specific side of things into roles on our side of things. It was an uphill struggle; these people weren't interested in it. Code didn't excite them, at all. They might as well have been shovelling fast food for all the love of craft they had. They came in at 8:30, got their work done, and went home at 5:30.

But the work they did? It would have bored me to tears. I would have quit had I had to do it; or perhaps, I would have surreptitiously built a framework or macro system or something to remove the redundancy and repetition from the tasks and indirectly reducing the tedium of the workload, rather than doing it more explicitly in what I was actually doing (developing the v.next framework for the company). But there's only so much redundancy you can squeeze out of many of these things (turning business rules from specifications into code) before you create too much complexity elsewhere. We still need lots of these people doing this boring work (most software development does not occur in software companies), and it's good that they're expert in the business rules they're translating. At the end of the day, it's OK that they're not great at producing code.

Of course, my opinions are completely invalid because they are generalized from limited experience, have no scientific value, etc. But it's the world I've lived, so I'd have to see decent studies and censuses to convince me otherwise.

[jdlshore]

That's the tricky thing about productivity, isn't it? You said that those domain-specific programmers were "not great at producing code," but I would argue (with tongue firmly in cheek) that in fact they were the productive ones, and you were not.

Let me explain. Productivity is defined as output over input. Programming input is activity--usually, time working--and output is software capability (or, more generally, business value). As architects, I'll bet that you didn't produce much that the company used. You probably produced designs, frameworks, or other tools for domain programmers to use. But the real software capability was produced by those domain experts you're criticizing, which means that they were the important actors and you were assistants at best. At worst, you could have been an impediment; I've seen "core" teams whose architecture-astronaut frameworks actually have to be subverted in order for the domain teams to get their work done.

I'm playing devil's advocate here, and I'm not entirely serious, but it's with a point: skill in coding isn't important unless it leads to tangible business results. In the short term, in my experience, business savvy outweighs coding skill.

(In the long term, technical debt becomes an overwhelming cost, so programmer skill does matter. But keeping technical debt low is more about maintainability, tests, and boringly understandable design than the kinds of things I see "rock star programmers" emphasize.)

[barrkel]

Please don't patronize me.

I tried to be specific in what I said; I said that it was OK, and good, that these people existed. I was not making an argument that these people are necessarily less productive; but rather, that there was a qualitative difference in how they approached code that made a difference in what they could accomplish with it.

(Though I think in the medium to long term, they will end up less productive, because of a lack of abstraction and leverage of existing code beyond copy and paste. Probably, we don't disagree that much.)

[jdlshore]

Sorry, I didn't mean to come across as patronizing.

[tetha]

At this point, the terms grow a bit wonkey though, and I am struggling to communicate this properly in upcoming interviews. The big, underlying question is: How productive is a programmer that enables other programmers to be productive. Especially if the enabling programmer works slower and only outputs things no customer will ever see and especially if the work done with the framework outweighs the time spent creating the framework by magnitudes?

[gruseom]

You pointed out that productivity can't be measured (though that hasn't stopped you from invoking the concept). Something else that can't be measured? "Business value".

Let alone the "business value" of an individual programmer's commits.

[jdlshore]

Yes, I completely agree. Why do you mention it?

[batista]

>Something else that can't be measured? "Business value".

If it makes money, it has business value, in my book. If it makes MORE money, it has more business value.

Actually it might be the ONLY measurable thing in the whole discussion.

[Retric]

Wait, so you have seen 10x variations in productivity but assume because 'bob' is say better able to avoid interruptions that does not mean he is a 10x better programmer? Programming is not just a question of time spent in front of a computer soft skills are important. If someone looks at inconsistent requirements and get's clarification rather than jump into a quagmire without end they are being more productive even if there not writing code at the time.

I think the 10-50x observed increase in productivity is task specific. But, people also deal with a finite list of specific problems so what if bob is 2x as productive on random tasks if there is 10 years worth of work where he can be 10x as productive he is worth a lot more than the average programmer.

[dkersten]

From what I heard the 10x meme came from a study back in the 60's where they timed something like ten programmers programming on punch cards for a couple of hours. Now, I didn't check the facts behind this myself, but if thats the case its hardly scientifically rigorous and hardly relevant at all today.

[dalke]

What you heard is incorrect. See http://forums.construx.com/blogs/stevemcc/archive/2011/01/09... for an incomplete list of some of the research which have been done, and which conclude that a 5x-25x variation exists.

[gruseom]

It's misleading to single out the 10x meme for the "that's not science" treatment. What beliefs about software development do hold up to the standard of hard (well-designed, controlled, and replicated) experiment? The published literature that I've read is all embarrassingly weak - understandably, given the complexity of what it's trying to study and the minuscule investment in such research.

I haven't read the ebook you're advertising, but a book with an identical pitch, Robert Glass' Facts and Fallacies of Software Engineering, is an example. The difference between his facts and fallacies is that the "facts" tend to have one poorly controlled, never replicated study (usually from 1978) to their name. Better than nothing (maybe), but close to nothing.

My point is that at this stage in history, it's all still folklore. Even you, immediately after playing the science card, followed with mushy folklore of your own ("I've certainly seen...").

Edit: come to think of it, your quotes make it sound like there is empirical evidence for the 10x claim, just that it's old and wasn't replicated. The "old" criticism is weird; it's not at all obvious that the internet invalidated the data. So the critique seems to boil down to: there was a study, but it wasn't bullet-proof and it wasn't properly replicated. What studies of software development isn't that true of?

[jdlshore]

I'm hardly "singling out" the 10x claim. If you knew me, you would know that I readily criticize programming studies [1]. At any rate, the prevalence of poor science in software development is no reason to be silent about this particular case of poor science.

I'll admit to being guilty of using anecdotes to describe my experiences with software development. (That's not quite the same as folklore, but whatever.) You won't find me claiming that my anecdotes prove anything, though, and I try to be clear that my perspective is based on experiences, not objective truth.

[1] For example, this comment three years ago on reddit: http://www.reddit.com/r/programming/comments/7dkjx/rescuing_...

[gruseom]

Fair enough. Is there any programming study that you think holds up as science?

[jdlshore]

Microsoft (Research?) published some stuff in the last year or two that looked really interesting. I don't have a link at the moment and I'm on my way out the door, unfortunately. But I remember thinking that it had promise because it drew conclusions from a huge base of data from actual Microsoft developers working on long-term projects.

Another paper I like is Little's report on data from projects at Landmark Graphics [1]. It's not a controlled study, and shouldn't be treated like one, but it also looks at a large base of data from real projects.

My biggest complaint about academic papers is that they're often conducted with students, using practices out of context, working on projects of trivial size and depth. I like both of the above not because they're perfect, but because they get beyond those problems.

[1] http://www.toddlittleweb.com/Papers/Little%20Cone%20of%20Unc...

[dalke]

I haven't read Bossavit's book. I have read McConnell's writings. Chapter 30 of "Making Software: What Really Works, and Why We Believe It" is by him, and edited by Andy Oram and Greg Wilson. The point of the book was to collect scientific evidence. I know Greg, and his high standards, so I strongly doubt that McConnell's evidence of weak.

I'm immediately suspicious of the statement you quoted: "Not a single one of the references is to a replication, in the scientific sense of the term." Strict replication is not and never has been a requirement to good science. As a trivial example, how do you replicate a supernova observation? You can't. But you can make models and test the models against the observed data and against future supernovas, and you estimate the reasonableness of the model effectiveness.

If that quote is indicative of Bossavit's views, then he is a poor judge of what's good evidence.

Going on with more quotes you gave, "which we can safely expect to have wrought major changes in programming practice." That is a conjecture, not a statement of fact. It seems like he's saying that once upon a time there was 10x difference, but the Internet makes everything different now there isn't. However, without evidence to back it up, I don't know why I should believe it.

What tests has Bossavit carried out to show that this is the case?

To the contrary, L. Prechelt "An empirical comparison of C, C++, Java ..." was both carried out during the internet era, was done with good rigor, and shows a large spread in the overall time to complete the project. That's a small project, granted, but it's another data point in the overall trends.

(If you don't like the small project size, another example, given by McConnell, is the large productivity difference between the Excel and Lotus 1-2-3 teams, which produced roughly comparable spreadsheet programs but with about an order of magnitude productivity difference across several different metrics.)

Multiple independently determined data points which all trend in the same direction strengthen the likelihood that the ~10x hypothesis is a useful model. That's what scientific evidence looks like.

McConnell even has a section in his chapter on "how meaningful it is to speak of an individual programmer's productivity", along with all of the caveats and cautions that everyone here is bringing up (some programmers produce negative lines of code, the best programmers tend to get the hardest problems, the effect of teams, etc.), and concludes "So while I see the value in measuring individual performance in research settings, I think it's difficult to find cases in which the measurement effort is justified on real projects."

Edit: I see that Bossavit wrote in response to McConnell's chapter. McConnel's reply to Bossavit is at http://forums.construx.com/blogs/stevemcc/archive/2011/01/09... .

[jdlshore]

Bossavit's book comes after McConnell's reply, and goes into more depth than the essay McConnell is replying to. Personally, I found the book more convincing. And having read what Bossavit had to go through to verify McConnell's references, I'm not convinced Greg Wilson did the same.

Rather than try to defend someone else's words, though, I'll just direct you to the book. You're going into a lot of detail critiquing quotes from an end-of-chapter summary, which isn't fair to Bossavit's work. I hope you'll read the book, because I'd like to hear your critique of the full work.

[dalke]

I would rather not. As it turns out, I haven't even finished reading "Making Software", not for lack of the quality of the essays but because my interest in this area isn't as high as I thought it would be.

I can be convinced otherwise for this one case. Could you elaborate for me on why my critique on these points is unjustified? Why is sufficient scientific evidence for to Bossavit? Why does the Internet necessarily discard older studies, and how does Bossavit establish that that's the case?

[rndmize]

Given that the idea of the 10x programmer seems to be considered bunk by most of HN (as per the above post), I'm curious about the idea of a Free Electron described here: http://www.randsinrepose.com/archives/2005/03/20/free_electr... . Would this just be another name for the same idea, or something separate that's more realistic?

[MortenK]

Looks at Spolskys article "Hitting the high notes" which is based on Yale prof. Stanley Eistenstat's meticolously gathered data of time spent among students for programming assignments.

http://www.joelonsoftware.com/articles/HighNotes.html