[pama]

I’m working in a related area and I’m rather curious about this point. In what way is GPT-4 overfit? Does overfit in this context mean the conventional: validation loss went up with additional training, or something special?

[rafiki6]

More specifically validation loss is irrelevant when you can't even sample out of distribution anymore.