Hacker News new | ask | show | jobs
by CaffeineLD50 459 days ago
A real user might be worse. A program is less flexible (maybe) and more consistent (definitely) than a meat space CBL.

The goal is not realism but a kind of ready made "you must be this tall to ride the rollercoaster" threshold.

Discovering edge cases with dodgy human users has its value, but that's a different value.
2 comments
gffrd 459 days ago
A real user will be worse … but that’s kinda the point.

The most valuable thing you learn in usability/research is not if your experience works, but the way it’ll be misinterpreted, abused, and bent to do things it wasn’t designed to.

link
cpeterso 459 days ago
Enter "Drunk User Testing". Host a happy hour event and give some buzzed users some scenarios to test.

https://www.newyorker.com/magazine/2018/04/30/an-open-bar-fo...

https://uxpamagazine.org/boozeability/

link
Tepix 459 days ago
More consistent? That's not a given with LLMs unless you set the temperature to 0.
link
CaffeineLD50 458 days ago
You are right. LLMs are totally random and useless.

Thanks for playing.

link
Tepix 458 days ago
You seem to disagree. Here's an interesting study where the researchers used an OpenAI-LLM-based tool to grade student papers and by grading them 10 times in a row, they got vastly different results:

https://rainermuehlhoff.de/en/fobizz-AI-grading-assistant-te...

Quote: "The results reveal significant shortcomings: The tool’s numerical grades and qualitative feedback are often random and do not improve even when its suggestions are incorporated."

link