[msapaydin]

>essentially a classification problem, with extra steps.

I think it is a question answering problem, with the extra of being able to say "I don't know", rather than a classification problem. The current transformers models for question answering such as huggingface pipeline implementations do not handle the "I don't know" case and this is an attempt to see whether gpt-3 can be trained to have this extra functionality easily.

>results are not universally consistent.

I think the relative probabilities are what matters and those may be more consistent than the absolute probabilities.