| HN Mirror

I remember someone doing a write-up, once, that nailed it. I won't be able to come close.

First, regardless of how well the group is chosen, it will never truly represent the actual user base. As long as that is kept in mind, while reviewing the results, that's great; but, in my experience, people tend to believe that the UG actually represents a valid proxy of the destination users, and aren't prepared, when the actual users get their hands on your baby, and start clicking on the wrong buttons, and not understanding something that's "perfectly obvious," etc.

Next, the group knows they are testing, and will do things like report an issue, instead of working around it.

That may seem to be the best thing, but the workarounds can tell you a great deal. Usually, it's discovering that the users have an entirely different mental model of the UX than you had thought. I've found that it can sometimes pay great dividends to "lean into" the "incorrect" mental model. It can make a night and day difference on how the product is received.

They may also not use it in their everyday work. At the company I used to work for, the UG was done on-premises, in a special room, with one-way mirrors, and cameras, all around. Users were given specific workflows, and did not use the software the way they wanted to.

We paid a lot of money for that, and it didn't tell us squat. It just gave us some "low-hanging fruit" feedback.

Also, it was my experience that managers and architect-types did not want to hear information that went counter to their own context. They would write off the results of user tests that indicated that their "baby" wasn't exactly a hit. This was about personal bias and ego, but, when the person writing the checks doesn't want to hear bad news, guess what? No bad news reaches them. If the product tanks in public, it gets a lot harder to ignore the bad news.

A well-done UGT can get things like this, but, in my experience, that doesn't happen.

Also, UGT is usually done under a veil of NDA/secrecy. Once the product hits the streets, people start hacking it, and, in my experience, these hacks can put my original work to shame. In one product I created, I worked for months to create some truly awesome clients for my backend. These were masterful. They answered all needs. They made the product great.

And no one used them. I used to receive almost constant bitching about my work.

Instead, a relatively "bad" programmer created a WP plugin, that everyone wanted to use.

Eventually, I threw in the towel on my client, and the new team that took over, made his plugin the reference implementation (including things like adding JSONP stuff that allowed it to be broken away from WP).

I am not a fan of sloppy MVPs, but there's a lot to be said for putting working code in front of actual users, in their "native" context, and then carefully observing them, in a non-Heisenberg way.

So, I guess TL;DR, is that I think controlled UGT is nice, but expensive, and of limited value. Open betas are probably better, as long as there is some way to solicit feedback on how the product is used.

Dealing with negative feedback sucks, but, in my experience, that is what is required to improve the product. I've received more value in profanity-laced diatribes, than in reams of Excel spreadsheets.