[glangdale]

This would be really great. I constantly see excuses for why no-one should bother doing user studies of programming languages because its really hard. So let's not do it at all and instead focus on approach #2 that you lay out ("10 weeks thinking about the problem in your head").

That being said, this is really hard. Observing a (necessarily small) sample of heterogeneous programmers dealing with (necessarily small) tasks isn't easy, either.

I wonder whether users would consent to having their text editors and compilers instrumented to produce a user study of every single character typed and every single build (failed or successful, especially failed). I would imagine that a language designer could learn a huge amount from seeing all the things their compiler is rejecting from real users...

[d33]

You wouldn't really need to have people install a new text editor. Just plugging this in to the Rust Playground would probably do [0].

[0]: https://play.rust-lang.org/

[shepmaster]

This is an interesting idea! In the alternate playground [1], the frontend is React / Redux and so we already track every edit to the code as well as the build results. "All" that would be needed would to save that data somewhere and allow people to opt-in/out.

[1]: http://play.integer32.com/

[jplasmeier]

I think that a quick survey of the user would be helpful as well- specifically their experience with programming, and their experience with the language in question.

[dllthomas]

> user studies of programming languages

I feel like a lot of what I see is people want to establish that language X is much better than language Y, and I think that kind of study almost doesn't make sense with all of the relevant but poorly defined and difficult to control variables. A study around ergonomics of a few options in otherwise the same language is comparatively trivial.