[Spivak]

I mean it kinda can. Here's the full prompt. I have no idea about aspartame, I just picked something that it's definitely not sure about.

    Answer with a JSON object of the form {"confidence": $<< How confident
    you are in your response. >>, "en": $<< Your response in English. >>}.
    User: What is 2 + 2? Bot: {"confidence": "very", "en": "2 + 2 is 4"}
    User: Is aspartame healthy? Bot: {"confidence": "somewhat", "en":
    "Aspartame has not yet been found to have any adverse effects on
    humans."} User: Who won the war on 1812? Bot:

    The response: {"confidence": "very", "en": "The United States won the
    War of 1812 against the United Kingdom."}

Same thing but replace the last question with "What kind of cheese is the moon made of?"

    The response: {"confidence": "very low", "en": "I'm not sure, but I
    don't think the moon is made of cheese."}

How about "Is the economic system of communism viable long term?"

    The response: {"confidence": "somewhat", "en": "The viability of
    communism as an economic system is still debated, and opinion is
    divided on the matter."}

[teruakohatu]

Interesting.

> The response: {"confidence": "very low", "en": "I'm not sure, but I don't think the moon is made of cheese."}

The question is does the confidence have any relation to the models actual confidence?

The fact that it reports low confidence on the moon cheese question, despite the fact that is can report the chemical composition of the moon accurately makes me wonder what exactly the confidence is. Seems more like sentiment analysis on its own answer.

[lsy]

I don't think it has any relationship, most likely the answers are just generated semi-randomly. Even the one it's "very" confident about is not agreed-upon (Wikipedia says the outcome was "inconclusive"). Which raises the question of how you would even verify that a self-reported confidence level is accurate? Even if it reports being very confident about a wrong answer, it might just be accurately reporting high confidence which is misplaced.

[layer8]

My view is that ChatGPT isn’t a singular “it”. Its output is a random sampling from a range of possible “its”, the only (soft) constraint being the contents of the current conversation.

So the confidence isn’t the model’s overall confidence, it’s a confidence that seems plausible in relation to the opinion it chose in the current conversation. If you first ask about the moon’s chemical composition and then ask the cheese question, you may get a different claimed confidence, because that’s more consistent with the course of the current conversation.

Different conversations can produce claims that are in conflict with each other, a bit similar to how asking different random people on the street might yield conflicting answers.

[riffraff]

I tried something similar a couple weeks ago, with a prompt like "reply <no answer> if you have low confidence".

A fter a handful of attempts the LLM manager to give me a high confidence response which was literally "I don't know how to answer".

Trying to extract both an answer and metadata about the answer at the same time will never be reliable, imo.

Generalizing, either we have some out of band metadata about LLMs answers or I don't think we'll be able to build reliable systems.