[trapper]

That's the same explanation psychics give. If the effect size is large (as is claimed), it should be an extremely simple study. So why has no one done it?

Why don't the lispers organize a RAD competition between lisp and all-comers, surely if lisp is that much more productive you would see the top ten places being lisp dialects.

Would you agree?

[tel]

Extremely simple in principle, obscenely difficult in implementation. The real difficulty is the confounding variables. Being unable to accurately measure or predict things like level of language familiarity, programming experience, domain knowledge, or intelligence means that it'd be impossible to tie any visible effects to something as simple as language choice.

So, no, I wouldn't agree.

The best study I could think of off hand would be a randomized study of programming productivity (measured somehow) of programming naïve high schoolers. Each subject gets an incentive to learn some language over the course of X weeks and then program something. The sample size would have to be magnificent even to prove a large effect because you'd need to demonstrate control for intelligence/drive and regressing out math scores (or anything similar) would not be convincing really. At the end, you'd still not be able to talk too much about working productivity, but you might have a chance of talking about... anything at all.

Actually, this is an extraordinarily difficult study I think.

[gridspy]

Perhaps high-school or college students are poor evaluators of a language. I am not only interested in how productive you are after 1 year of using a language, I'm also interested how productive you are after 10 years using that language, or with 10 years of legacy code.

[tensor]

Well, I'm not sure a RAD competition would necessarily be considered scientific. For one, the sample size is small. It also ignores properties such as maintainability, assuming "first to finish" is your only criteria. Finally, it's necessarily a fairly small application I would think.

As for studies, a quick search on google came up with this: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.117...

Which concludes that "Haskell prototype took significantly less time to develop and was considerably more concise and easier to understand than the corresponding prototypes written in several different imperative languages, including Ada and C++."

I couldn't find much more in a short time. Perhaps my search terms, "functional programming imperative productivity" were inappropriate.

[trapper]

Now that is interesting. I personally agree with the hypothesis that lisp is a step forward, but often wonder why it's an anecdotal argument than one that is data backed.

This paper although makes me think they compared programmers not languages, as the imperative language programmers didn't have any functional experience. It would be interesting to see a comparison between programmers of roughly equal competency.

[tensor]

Mostly it's due to a lack of good, controlled, data. The ideal study would need to be of a large and complex system that was recreated several times in several languages. You'd need to verify each system behaved the same, and that the teams on each version were nearly equally competent in their respective languages. Even that would be poor, since it would be a single data point. Perhaps a particular language was simply good at that problem!

Despite that complaint, the above is nearly impossible to do in practice. So you have to settle for smaller problems, which can be done in several months perhaps. Even at that, it would take some very difficult collaboration, and would still be a single data point.

Further complicating issues is the academic world vs. industry. Again, ideally you'd want to do the test in industry, with an appropriate team of programmers and managers. But industry doesn't want to waste time with such academic matters, nor does it want to open it's code for analysis.

In other words, it's a very difficult problem to address, despite being seemingly simple on the surface.

[tel]

This paper is another interesting anecdote, unfortunately. They're not really claiming anything quantitatively.

Also, though it's not explicit, I feel it's obvious that the researchers had a prior bias toward functional languages.

[camccann]

A competition to solve a toy problem wouldn't necessarily say much, though. So the question is, how big of a problem would you need, and where are you going to find enough programmers (of reasonable and roughly similar competence) who don't have anything better to do than spend multiple hours participating in a glorified programming language pissing match?

In other words, the reason why not is probably "because organizing things is hard work".

[trapper]

If the effect size is large (i.e. the magnitude of difference between the means of lisp vs non-lisp programmers) the size of problem should be irrelevant.

Many thousands of programmers spend hours each week in pissing matches online. Why is it none of them feel like truly putting their money where their mouth is?

[camccann]

If the effect size is large (i.e. the magnitude of difference between the means of lisp vs non-lisp programmers) the size of problem should be irrelevant.

Writing a "Hello, world" is going to take about the same amount of time in any language that isn't a turing tarpit. At issue is generally some sort of asymptotic advantage, not a guarantee of "language X will always be superior every time always". In fact, for any pair of one easy and one hard problem, I'm pretty confident I could find (or invent, if necessary) two languages that each beat the other for solving one of the problems.

The claimed benefits of a more expressive language generally involve things like better management of application complexity, or greater ability to modify and restructure late in the project. Hammering out a program of a scale such that the programmer can keep an accurate model of the entire thing in mind at one time, and that is only written and never modified, would be more a measure of the programmer's typing speed and the sophistication of the IDE used than anything else.

[prodigal_erik]

If a problem is so trivial that creating a domain-specific language isn't worth the effort, the best solution will be equally ugly in any language and Lisp loses its advantages.