[espadrine]

Given that Mr Yudowsky is not the only one to have won the game, it seems, from however little leaked, that what made them win isn't rational.

If you have ever cried or been scared while watching a film, you may know that you can be affected by things you don't experience directly. However, making a film that affects you requires a lot of skill, preparation, and analysis of the target audience.

The whole point of the game is for the AI to scar you emotionally, until you do something you'll regret. It is as trying to the subject as it is to the (human-played) AI, especially considering that not only is it unethical, it requires to act unethically extremely fast, similar to how a jump scare works—the least you expect it, the better it works.

It is not something I wish on to any unprepared individual. It is also not something anyone would expect to happen from a "game", which is probably why Mr Yudowsky won so many times.

But the real question is not "how would anyone react to a smarter AI in a box". We all know from Milgram's experiment that anyone can be driven to do unspeakable things. The real question is "how to train someone against an AI in a box".

[FeepingCreature]

The real question is "why do people think it's a good idea to create a superintelligence of questionable morality and have the only defense against it be a _human gatekeeper_". And yes, this notion was seriously proposed.

Eliezer does not want to put AIs in boxes. He thinks the entire idea is hopeless; _hence the game_.