[lysozyme]

Do yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT

>For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The core issue is that we want to encourage the model to answer based on its internal knowledge, but we don't know what this internal knowledge contains. In supervised training, we present the model with a question and its correct answer, and train the model to replicate the provided answer.

The author says he’s summarizing a talk by John Schulman of OpenAI [2] but I haven’t personally watched the video. In any case, this is an interesting insight.

Say we set up a supervised learning scenario where we ask the model to use its internal knowledge to answer a question and compare its answer to one written by a human. If the two answers essentially say the same thing, but in different words, in the supervised learning case the model is penalized. In the RL case, it’s rewarded. That’s the difference.

1. https://gist.github.com/yoavg/6bff0fecd65950898eba1bb321cfbd...

2. https://www.youtube.com/watch?v=hhiLw5Q_UFg

[laprise]

Yes indeed, Yoav Goldberg's post is essentially a good summary of John Schulman's talk, which is excellent. I highly recommend people watch it.

However, my point goes beyond this technical argument. I would argue that even if, by some magical process, we could perfectly replicate GPT-4 behaviour, I still don't think it's a good idea or at least it's not enough. Don't get me wrong, it would be really handy to have a free version running on our own cluster, but it wouldn't fix the other issues I mentioned.

[gojomo]

Yes, the inner link to the Yoav Go writeup is the gem here, concisely explaining the benefits of RLHF-scoring over mere prediction.

Though, as Go speculates, it's likely possible to reduce even further the HF ("human feedback") part, while still reshaping the model to have the helpful qualities.

My guess is there's a rich set of potential ways to this – automate that extra level of distinction between mere "exact token prediction" & "sufficiently valuable responses" – & OpenAI probably has a few undisclosed advances here as part of their GPT4 training/tuning.

In particular, Go's suggestion that a separately tuned LLM can do a fuzzier scoring of whether an answer is "close enough" to an idealized answer seems like the sort of promising ensemble approach that will have been an obvious next step for most LLM teams, probably being tried by many independent teams right now.

[b33j0r]

It seems pretty transparent, as I think you might be implying in part, that attempting to leap-frog without directly copying training data or model weights is a temporary optimization for “bootstrapping” teams.

I see this pretty directly in stablelm releasing both a base model, and a tuned model… which is not based on the base ;)

There is a goldrush to get training sets worth using, and if something 90% quality gets your models on the map quickly, it’s an attractive option. As they say, attention is all you need.

Training in the 7B range is a lot cheaper than I expected. Fine-tuning almost negligible—if you have clean data, which has always been the expensive part.

Most humans expect more than $0.000003 per token as compensation for _your_ dataset collection.

[avereveard]

I want to add an argument:

I hate gpt style of "as an ai model I can/can't" answers, any model distilled from that corpus becomes very hard to use for tasking.

Like you may just want the category of a text, but all your equals now become contains. It eats up a lot of token space. It begins as a sentence so categories now are strongly biased toward sentence case and not your original input.

I know at least one model purposefully removing these utterances, but still. Everyone else is chasing the agent feel, and I'm here pulling my hair out because prosumer were this close to be able to access a proper AI for tasking and now it's slipping away.

[rvnx]

Especially, it's not that it can't give you an opinion, it totally can, but the authors at OpenAI don't want you to hear that specific opinion because of their political biases.

[pixl97]

In converse, the authors at OpenAI don't owe you a service that caters to your own political biases.

[ChatGTP]

Lol yet they enjoy a monopoly on the industry so there are no alternative world views as far as LLMs are concerned

[pixl97]

They do?

Google doesn't have their own LLMs? Microsoft isn't running and training their own copies?

Hell, if you have enough money you could run your own. This doesn't sound like a monopoly in any definition that I use (and I cast a very wide net with my use of monopoly).

[ChatGTP]

Name one company which has an product as popular as ChatGPT-4?

[laprise]

yes exactly, this "as an ai model I can/can't" you're mentioning is precisely what is learned during the RLHF alignment phase.

[laprise]

Btw, this kind of language is already all over the place: https://twitter.com/d_feldman/status/1649466422018318338 Which is hilarious and sad at the same time.

[avereveard]

It is a funny tweet but I cannot replicate the results.

[blueblimp]

Schulman's talk is great. I had been thinking about this problem, and he covers almost everything I thought of, with great clarity.

One thing he didn't mention though is that there's potentially a bit of a trick to get the fine-tuning datasets to transfer across models anyway. (I haven't tested it.)

The key idea is to eliminate the pronouns. Imagine asking GPT-4, not whether it knows a fact, but whether "gpt-3.5-turbo" knows, or "text-davinci-003" knows, etc. Then, when you want the model to reply using pronouns (e.g. "I don't know"), use the system message to tell it which model it is.

This doesn't benefit from introspection, so quite possibly it doesn't work. The reason it might work anyway, though, is that estimating the difficulty of a question might be possible even without introspection.

[jimsimmons]

In RL case it’s rewarded because the supervision signal is generated post hoc.

You can do the same with pure supervised learning and no RL. HF is the key, not RL.

Yoav misses the nuance John had. RL is not bringing something fundamental to the table. It is just a better way to do things at the moment