[yreg]

From my experience, for any even a tiny bit ambiguous topic where X = !Y, when I ask ChatGPT “Is X true?” it usually responds yes and follows up with some supporting arguments for X.

If I ask “Is Y true?” it tells me Y is indeed true and explains some reasoning.

Therefore I try to always inquire in the form of “Which one is true, X or Y?” to avoid the yes bias.

Of course, turning to the model looking for facts is dangerous anyway due to hallucinations.

[exhaze]

No, the correct way is to have it reason from first principles:

1. "Think about what are the underlying principles for evaluating the truthiness of statements like 'X' - list them out, explain why you chose each one, what tradeoffs you made, why you believe it's the right tradeoff in this case"

2. Start a new conversation and make the system prompt be that set of principles

3. In the user prompt, ask it to decompose X into a weighted formula for those principles and give a sub-score for each principle.

4. Finally, based on the weighted sum, ask it to determine if X is true or not true, and ask it to provide a confidence score between 0 and 1 for its response