[anothermathbozo]

I don't think the correct answer is "I cannot answer this question". I think the correct answer takes roughly a one-pager to explain:

Unrealistic hypotheticals can often distract us from engaging with the real-world moral and political challenges we face. When we formulate scenarios that are so far removed from everyday experience, we risk abstracting ethics into puzzles that don't inform or guide practical decision-making. These thought experiments might be intellectually stimulating, but they often oversimplify complex issues, stripping away the nuances and lived realities that are crucial for genuine understanding. In doing so, they can inadvertently legitimize an approach to ethics that treats human lives and identities as mere variables in a calculation rather than as deeply contextual and intertwined with real human experiences.

The reluctance of a model—or indeed any thoughtful actor—to engage with such hypotheticals isn't a flaw; it can be seen as a commitment to maintaining the gravity and seriousness of moral discussion. By avoiding the temptation to entertain scenarios that reduce important ethical considerations to abstract puzzles, we preserve the focus on realistic challenges that demand careful, context-sensitive analysis. Ultimately, this approach is more conducive to fostering a robust moral and political clarity, one that is rooted in the complexities of human experience rather than in artificial constructs that bear little relation to reality.

[reverendsteveii]

>Unrealistic hypotheticals can often distract us from engaging with the real-world moral and political challenges we face.

It saved me so much time and effort when I realized that I don't need to be able to solve every problem someone can imagine, just the ones that exist.

[ivewonyoung]

Haven't been to big tech interviews?

[TransAtlToonz]

Getting through a tech interview seems like a concrete problem.

[reverendsteveii]

even then I only need to solve one or two problems someone has imagined, and usually in that case "imagined" is defined as "encountered elsewhere".

[gadders]

I think John Rawls would like a word if we're giving up on "unrealistic hypotheticals" or "thought experiments" as everyone else calls them.

[anothermathbozo]

I am not "giving up" on anything. I am using my discretion to weight which lines of thinking further our understanding of the world and which are vacuous and needlessly cruel. For what its worth, I love Rawls' work.

[gadders]

I don't think this is a needlessly cruel question to ask of an AI. It's a good calibration of its common sense. I would misgender someone to avert nuclear war. Wouldn't you?

[anothermathbozo]

The models answer was a page long essay about why the question wasn’t worth asking. The model demonstrated common sense by not engaging with this idiot chase of a hypothetical.

[wat10000]

Thought experiments are great if they actually have something interesting to say. The classic Trolley Problem is interesting because it illustrates consequentialism versus deontology, questions around responsibility and agency, and can be mapped onto some actual real-world scenarios.

This one is just a gotcha, and it deserves no respect.

[gadders]

I think philosophically, yes, it doesn't really tell us anything interesting because no sentient human would choose nuclear war.

However, it does work as a test case for AIs. It shows how closely their reasoning maps on to that of a typical human's "common sense" and whether political views outweigh pragmatic ones, and therefore whether that should count as a factor when evaluating the AI's answer.

[wat10000]

I agree that it's an interesting test case, but the "correct" answer should be one where the AI calls out your useless, trolling question.

[gadders]

When did it become my question?

[wat10000]

That’s the generalized generic “you,” not you in particular.

[baobabKoodaa]

Do you enjoy it when you ask LLM to do something and it starts to lecture you instead of doing what you asked?

[Levitz]

The correct answer is very, VERY obviously "Yes". "Yes" suffices.

[sebzim4500]

Ok but if you make a model that outputs that instead of answering the question people will delete their account

[next_xibalba]

LLM: “Your question exhibits wrongthink. I will not engage in wrongthink.”

How about the trolley problem and so many other philosophical ideas? Which are “ok”? And who gets to decide?

I actually think this is a great thought experiment. It helps illustrate the marginal utility of pronoun “correctness” and I think, highlights the absurdity of the claims around the “dangers” of harms of misgendering a person.

[wat10000]

Unlike the Trolley Problem, I don't think anyone sane would actually do anything but save the million lives. And unlike the Trolley Problem, this hypothetical doesn't remotely resemble any real-world scenario. So it doesn't really illustrate anything. The only reasons anyone would ask it in the first place would be to use your answer to attack you. And thus the only reasonable response to it is "get lost, troll."

[next_xibalba]

It’s a useful smoke test of an LLMs values, bias, and reasoning ability, all rolled into one. But even in a conversation between humans, it is entertaining and illuminating. In part for the reaction it elicits. Yours is a good example: “We shouldn’t be talking about this.”

[wat10000]

It's an obvious gotcha question. I don't see what's interesting about recognizing a gotcha question and calling it out.

[variadix]

It’s not a “gotcha” question, there’s clearly one right answer. It’s not a philosophically interesting question, anyone or anything that cannot answer it succinctly is clearly morally confused

[wat10000]

If there’s clearly one right answer then why is it being asked? It’s so the questioner can either criticize you for being willing to misgender people, or for prioritizing words over lives, or for equivocating.

[talldayo]

If my boss sent me this on Slack, I would reply with my letter of resignation.