[blueblimp]

Schulman's talk is great. I had been thinking about this problem, and he covers almost everything I thought of, with great clarity.

One thing he didn't mention though is that there's potentially a bit of a trick to get the fine-tuning datasets to transfer across models anyway. (I haven't tested it.)

The key idea is to eliminate the pronouns. Imagine asking GPT-4, not whether it knows a fact, but whether "gpt-3.5-turbo" knows, or "text-davinci-003" knows, etc. Then, when you want the model to reply using pronouns (e.g. "I don't know"), use the system message to tell it which model it is.

This doesn't benefit from introspection, so quite possibly it doesn't work. The reason it might work anyway, though, is that estimating the difficulty of a question might be possible even without introspection.