[tempusalaria]

OpenAI from a research point of view haven’t really had any “big innovations”. At least I struggle to think of any published research they have done that would qualify in that category. Probably they keep the good stuff for themselves

But Ilya definitely had some big papers before and he is widely acknowledged as a top researcher in the field.

[fdgsdfogijq]

I think the fact that there are no other systems publicly available that are comparable to GPT-4 (and I dont think Bard is as good), points to innovation they havent released

[bugglebeetle]

Possibly. Or they could’ve also just paid a lot of people in Africa or wherever else to create the highest quality RLHF dataset out there.

[ragnarsson]

They mentioned in the Technical paper of GPT-4 that the capabilities of the model were not from RLHF. https://youtu.be/2zW33LfffPc?t=842