[bluGill]

Human factors studies on programming languages are really hard to do right. It is easy to study someone seeing a language for the first time. However programming well requires a high level of expertise and so the real question isn't how easy it is for a beginner, it is how easy it is for someone who has been doing it for years. Or maybe how much different is someone after a week vs month vs year (that is at what point do the experience gains plateau)- in all this you have to be careful to ask different tasks - since good programmers will abstract when they need to do something twice.

[Rochus]

I think it's not more difficult than identifying representative user groups (at least five members per group) and defining typical tasks for a usability test, focussing on the specific features in question. It's just a lot of work and requires experience and training.

[kthielen]

You make a very good point, and maybe a problem here is that “human factors studies” are set up like market research rather than anthropology.

People who’ve spent a long time programming have spent a long time optimizing everything about their work, and they’re willing to talk about it (most of them won’t shut up about it, even).