[modeless]

Are there plans to do user studies? 10 minutes watching new users code in Rust will give you better ideas than 10 weeks thinking about the problem in your head.

I feel like there's a real lack of user testing in software development tools land. If you're developing software for unsophisticated users it's obvious that you should be doing user testing, but it's an often ignored fact that developers are users too! APIs, compilers, build tools, they could all be vastly improved with some user testing.

[hackuser]

> 10 minutes watching new users code in Rust will give you better ideas than 10 weeks thinking about the problem in your head.

I'd add: Do you want to focus on new or experienced users? For example, when implementing systems that will be used 8 hrs/day by its users, we look only at efficiency for experienced users (unless the political situation requires placating the noobs). They will be noobs for a day or maybe a couple weeks; they'll be experienced all day, every day for years. We explain it that way at the beginning and they thank us later.

An example many are familiar with is Vim: Steep learning curve, but I'm so happy that the focus is on efficiency for veteran users.

[lemming]

This is the reasoning that is used for Clojure - "Clojure is for experts". It's used to rationalise all sorts of design choices that make the language much harder to get started with than (IMO of course) it should be.

If you're implementing a system that users have to use for their job, they'll put up with whatever they have to while they're learning it. If you're implementing a programming language and you want to increase adoption and general user friendliness, I think this is a terrible choice.

I love Clojure now and I work with it every day, but it's still frustratingly hard for avoidable reasons even though I'm a pretty advanced user. I'm jealous of the attention that Rust, Elm and Elixir pay to this aspect of language design, and I'm very pleased to see languages taking it more seriously.

[sambe]

I guess the optimum has to be some kind of "area under the learning curve" but I think it's pretty clear if you made a binary choice optimising for people who already made the effort makes a lot more sense. Is Clojure really that bad? I don't know it, but I never had the impression there was an anti-newbie community. Sometimes people take these binary positions as a decision-making optimisation too; having that policy just simplifies a lot of discussions, perhaps.

[throwaway7645]

Agreed. All of those were easier to deal with than Clojure, but that could just be because it is a pain to do anything with the JVM and emacs isn't easy either.

[nathancahill]

What's the language that takes that to the extreme? I remember it being used for fintech(?) or spreadsheets and I remember seeing one-liners that look like someone just mashed the keyboard. Apparently incredibly efficient once you are an expert in the language.

[throwaway7645]

The APL family of languages that currently have commercial implementations like Dyalog and Q w/ kdb+ database. J and GNU APL are both free & open source. Dyalog is nice though even though a commercial license sets you back a grand per year (basically what most C# users pay for VS). You get a nice interpreter, support, built-in graphics, full .NET interop, full R interop, heck even DDE which is still useful in finance and my own industry. They seem to take documentation really seriously and have pretty interesting conferences. I don't code in APL, but am considering it as their product just seems pretty agile for my needs (personal user productivity). Q is extremely expensive and is the APL derived language that accompanies the kdb+ database used for time-series analysis on stock quotes...etc. It was built by Arthur Whitney and legend has it the entire source code is 5 pages of C (being an APL guy he has it all scrunched up though...doesn't like to scroll). People seem to like it and be willing to pay a fortune.

[mickronome]

That would probably be Q used in kdb+, and J, which the previous response stated is an open source implementation of the same/very similar language. The terseness even visible in the language name.

[hyperpape]

Perhaps J, which is part of the APL family of languages.

[falcolas]

I think that you should focus on both groups. The new users for adoption, the advanced users for retention.

Even if no actual language changes come out of the new users group, just documentation updates, it would be a win for the entire ecosystem.

[coldtea]

>unless the political situation requires placating the noobs

If something is a pain point for the noobs it's usually a badly designed part of the language that the "veterans" have just learned by rote to step around.

[glangdale]

This would be really great. I constantly see excuses for why no-one should bother doing user studies of programming languages because its really hard. So let's not do it at all and instead focus on approach #2 that you lay out ("10 weeks thinking about the problem in your head").

That being said, this is really hard. Observing a (necessarily small) sample of heterogeneous programmers dealing with (necessarily small) tasks isn't easy, either.

I wonder whether users would consent to having their text editors and compilers instrumented to produce a user study of every single character typed and every single build (failed or successful, especially failed). I would imagine that a language designer could learn a huge amount from seeing all the things their compiler is rejecting from real users...

[d33]

You wouldn't really need to have people install a new text editor. Just plugging this in to the Rust Playground would probably do [0].

[0]: https://play.rust-lang.org/

[shepmaster]

This is an interesting idea! In the alternate playground [1], the frontend is React / Redux and so we already track every edit to the code as well as the build results. "All" that would be needed would to save that data somewhere and allow people to opt-in/out.

[1]: http://play.integer32.com/

[jplasmeier]

I think that a quick survey of the user would be helpful as well- specifically their experience with programming, and their experience with the language in question.

[dllthomas]

> user studies of programming languages

I feel like a lot of what I see is people want to establish that language X is much better than language Y, and I think that kind of study almost doesn't make sense with all of the relevant but poorly defined and difficult to control variables. A study around ergonomics of a few options in otherwise the same language is comparatively trivial.

[rattray]

On this note, I'm actively seeking JavaScript developers interested in doing some user studies of a compile-to-js language I'm working on.

It emphasizes ergonomics, actually, as well as making functional best-practices (typed immutability, etc) more accessible.

I'm in San Francisco but happy to conduct remotely. I'm targeting people familiar with ES6 and functional styles, but interested in folks of all experience levels.

If anyone would be interested in trying it out, shoot me an email at rattray.alex – gmail.

[girvo]

Replying to chase this up tomorrow when I'm not on my phone

[steveklabnik]

We have done informal ones and may or may not do more formal ones, we'll see.

[modeless]

Informal is fine. The important thing is to do user testing in an ongoing way, not just once. Without a feedback loop it's quite surprising how quickly development priorities diverge from what's really important.

[kibwen]

Members of all the Rust subteams are constantly involved in guiding new users through the language. Come to any Boston Rust meetup and you'll find Niko Matsakis giving one-on-one help to people taking their first steps. And it's not just feedback from people in major tech hubs; Brian Anderson and Alex Crichton just got back from mentoring new users at Hack Illinois in Urbana. They go to great lengths to remain plugged-in like this.

[steveklabnik]

We are always constantly trying to listen to people and get feedback; for example, it's one of the reasons I pay such close attention to these threads!

There's always room for improvement, but iterating in this way is defintely a core value.

[IanCal]

Depending on what group you're trying to attract or improve the experience with, you could potentially setup contracts with the level of person you would like to deal with and pay them to implement something in rust while you monitor their progress & with full access to their work and what they're doing.

[vertex-four]

The issue is, of course, that the more you focus on specific tasks the more your language becomes "designed for" those tasks, and the language becomes less general. The language already went through some of this with Servo - the DOM would be much easier to implement with an object-oriented language, but that wouldn't fit in well with the rest of Rust and would significantly raise complexity.

[IanCal]

That is a concern but the other risk is by not doing this the language progresses in areas people happen to currently use it, which will be skewed by what it's already good at. You can then end up never improving for those cases it's weakest. This covers types of programs but also experience levels and histories of the programmers themselves.

You are right however that you need to carefully pick the tasks to fit with the original aims of the language.

Perhaps a good way of phrasing this is "why aren't X people using rust for Y?".

[zzzcpan]

"If you're developing software for unsophisticated users it's obvious that you should be doing user testing"

Compilers actually have a unique opportunity to introduce some form of opt-out tracking of all of the compilation errors with sources, etc. This could really help to understand the users.

[Animats]

No, that's not how it's done. Proper testing for computer usability involves video of the user and the screen. The important things to note are when the user stops, gets stuck, has to consult external references, asks for help, or gets upset. It's far too intrusive to impose on anyone not explicitly volunteering to do it.

[spangry]

Unfortunately I have but only one upvote to give. This is very important if you're performing a usability / ergonomics study. Context matters a great deal. The parent poster is also quite correct about it being invasive: although not absolutely necessary, ideally you'd observe the user in whatever context they happen to use Rust (e.g. at their workplace or in their home). Lab studies are still valuable too, but you might miss some context due to the artificial setting.

[steveklabnik]

We have discussed doing this in the past, actually. Opt-in (oops, had "out" here initially) is very important here!

I am not sure if it's something that ever landed or not...

[IanCal]

Opt-in tracking, I really hope you mean. I know the concerns with getting unrepresentative samples but sending what I'm working on to a third party is something that would immediately stop me using an entire language for any form of my work.

[steveklabnik]

gah! Yes, I mean "we would never consider tracking you without your express given consent." That's opt in not opt out.

[IanCal]

:) Thought that was the case, seemed odd that people were saying it like it was such a small deal!

[zzzcpan]

It doesn't have to be strictly opt-in or opt-out. There could be two versions released, a version without tracking could live on the same release page, just a bit lower and not in a bold font or something like that, pushing new users towards the version with opt-out tracking.

[rossng]

This is a dark pattern. Why would you want the default to be a compiler that essentially spies on your code? I can think of no better way to make developers (not to mention businesses!) uncomfortable with the idea of adopting Rust.

Thankfully, Mozilla is one of the few entities left in the technology industry that seems to care about ethics. I don't believe the Rust team would ever do something like this.

[zzzcpan]

No, it's not a dark pattern, you have a clear choice, just one being less advertised than the other one.

And Mozilla is the one with opt-out of all of the unethical features in Firefox. They have no ethics, looking like they care about those things is nothing more, but PR. After all, corporations are not people.

[arjie]

Is there a mailing list I can subscribe to that will tell me when I can turn this on? I'm hoping not to have to listen to rust-dev for the trigger phrase.

[steveklabnik]

I don't know off the top of my head because it's been a while since we talked about it and I haven't paid super close attention to rustup's development. I am assuming that we would make an announcement on the blog because we'd want to be very clear about what's going on, and of course, try to convince people to opt in :)

[deegles]

How about opt-out on nightly builds and opt-in on production?

[steveklabnik]

I don't personally think opt-out is ever acceptable for this kind of thing.

[glangdale]

My comment crossed with the GP post here, but that's a really good idea. Please look into it!

[Too]

With good auto completion, the user should never even have to press the compile button or hit a compiler error. (Slight exaggeration)

This means the user might have struggled a long time with the code before even hitting the compile button. If you only collect statistics from compilation you will miss all this pre-struggle.

[bsaul]

Was about to post the same idea. This should most definitely be the number 1 step for the whole "simplification" project. Take all the advice on how to build a start up by focusing on data points, and apply it to a programming language.

[AsyncAwait]

> 10 minutes watching new users code in Rust will give you better ideas than 10 weeks thinking about the problem in your head.

I am not sure whether 10 minutes of Rust would give you actionable info, I mean it certainly takes longer to get familiar with the basics in almost any language, even if the 10 minutes just stands for a short amount of time.

[pekk]

Things happen even in the first 10 minutes.

It's not appropriate to limit user studies only to people who are already experts.

[dllthomas]

But it's also questionable to limit them to people who are not, depending on what you want to accomplish.

[falcor84]

I assumed "new users" implied relatively new users (before they grew to love the language enough to accept the downsides), not absolute beginners

[lomnakkus]

Although I think I agree that many current languages do not pay enough attention to ergonomics (hello, Scala!), I think your line of reasoning could be quite dangerous.

I don't think there's any empirical support for the idea that a shallow learning curve translates into a powerful/expressive language. EDIT: I'd love to be proved wrong, so if there's anything in the way of evidence, please link me!

[acjohnson55]

As a Scala dev, I'm amused to see the Scala call-out :)

Undoubtedly, it's a language that is a little tricky to get fully fluent, but I'm curious what things pop out to you as bad ergonomics? I think it actually doesn't suffer in many of the ways identified for Rust in this article.

[lomnakkus]

It's a bit late here, so I won't go into much detail, but just rant a bit. Hope that's OK and gives a general impression of my personal gripes.

EDIT: Well, that rant turned out longer than I imagined... and it's even later now, so I'll probably go to bed momentarily.

EDIT#n: PS: Scala's still better than Java, but please don't tell me Scala's in any way "coherent" or "well-designed" or "orthogonal". Haven't really tested Kotlin or Ceylon, but I get the feeling that those are really much more c-or-w/d languages for the JVM. Plus, "orthogonal" doesn't mean shit if your ergonomics suffer like they do in Scala.

First: See Scala Puzzlers + PaulP's rants... and that's not even half of it. (Yes, I know collections are going to be "fixed", but... not really. I don't think the fact that Odersky still seems to be calling all the shots on that front is in any way encouraging.)

Second: An especially annoying thing is the way "_" desugaring works which sometimes leads to really surprising results when it turns out that "no, you cannot use _ to stand for 'identity' in a function parameter context". This was a conflation of 'features' namely that scalac doesn't warn when a unit-returning method is actually with an expression that doesn't return unit. If that explanation doesn't make sense, I apologize -- it's a complicated issue that's hard to explain... which is exactly my problem with some of the problems I've experienced in Scala :)

Third: The whole "let's encode typeclasses as implicit parameters" thing is insane. AFAIUI this is actually a thing that was intentional. Unless you have actualy real+usable dependent types it's a huge mistake to get rid of coherence[1]. (Plus, TCs are a semantic-level feature. You don't want to force people to "encode" semantic things you care about in your concrete syntax. That way lies "[OOP] patterns".)

Fourth: The whole "for-yield" thing. WTF? Either do monadic notation like it matters or just don't do it. For-yield leads to incredibly noisy code. (It's not even consistent, try starting your for-yield with a "x = ..." line, and you'll see.). The fact that monadic notation starts with a "for" and that collection iteration starts with a "for" is NOT in any way a sign that your language design is "orthogonal" as Odersky loves to claim. It just means that the language designer didn't really understand what he was doing. (As you can probably gather I'm a Haskeller/Idirs'er/etc.). Syntax for these things matters.

Fifth: ... and a follow-up to that: @tailrec isn't enough: That fact that explicit trampolining is basically required for any monadic Scala code basically kills the idea of monadic programming right off the bat... unless you're willing to invest huge amounts of time into it for-yield is just unusable. (Yes, I'm aware of the Free monad. One hopes that the performance problems have been solved, but it's still not always enough to have a Free monad.). Honestly I think it would have been better to invest time into implementing tail-rec on the JVM rather than wasting that time on @tailrec. I do mean wasting -- any situation where @tailrec applies, I could have trivially replaced (or "encoded" that "pattern") that code with a while loop and a var.

Sixth: Why isn't Monad/Monoid/etc. in the standard library? This goes back to my third point which is that the language doesn't actually encode semantics of typeclasses, and so now we have "scalaz" with its Monad and "non" with its Monad, and "fs2" with its scalaz-stripped-down-copy-of Monad. WTF? This is not needed -- these typeclasses have stood the test of time and it's only because Scala insists on encoding them that we ended up here. (Encoding via various implicit magic tricks, the fact that everything has to be an object, etc. It's fucking ridiculous.)

Summary: There's a lot of completely accidental complexity because Scala wasn't really designed as a coherent whole... it just sort of "grew organically". (Like almost every language out there.)

[1] Yes, you gain a "choice" of instances via imports. Guess how often that's actually useful vs. harmful? Never. It's absolutely insane that your imports can change your serialization/deserializion code.

[jeffdavis]

What programming languages have been developed in this manner? What were the results?

[spenrose]

Python. It was lauded for usability and became a 30 year success: http://python-history.blogspot.com/2009/02/early-language-de...

[zzzcpan]

None. Typical usability testing with watching people do things doesn't apply to programming languages, because of how much more complex and larger languages are in comparison. At best this approach can be used to test usability of tooling, IDEs, but that's it.

[nathan_long]

I've heard it said that the creator of Elm regularly pairs with newbies and adjusts the language based on what they find difficult, but I don't have a citation.

[pjmlp]

If I recall correctly Microsoft did some regarding VB.

[tatterdemalion]

When we're trying to balance a trade off, I often wish we could just do A/B testing on it, but that's not realistic for languages.

[nostrademons]

What you can do is run over a large corpus of programs (crates.io? GitHub?) to identify the places where the different alternatives would impact code, collect metrics on them, and pull some sample programs to investigate the impact on them of the different alternatives. Is anything like that being done?

The HTML5 spec was developed in this way, and some of the code-health folks at Google were just starting to operate this way when I left, but I haven't heard of many other language-design efforts working like this. The technology & data exists now, with large public corpora of open-source code, parsing libraries that can identify patterns in it, and big-data tools to run these compiler tools over a large corpus. It's a bit of a paradigm shift from how languages are typically developed, though.

[mattyb]

Russ Cox did this when figuring out how to seamlessly add monotonic elapsed time measurements to Go:

https://github.com/golang/proposal/blob/master/design/12914-...

[tatterdemalion]

We use crater to check potentially breaking changes against our entire package ecoystem, but its not as easy to test which of two syntaxes for a new feature will be easier to use by looking at existing code.

[joseraul]

They may not do user studies like this, but for sure they adapt to user feedback.

https://thefeedbackloop.xyz/stroustrups-rule-and-layering-ov...

[sametmax]

God yes. If anybody must remember one concept only about ergonomics, it's that.