[politician]

The article points out that training data generated using ChatGPT is necessarily biased or tainted with the consequences of the policy optimizations and RLHF alignment processes conducted by OpenAI. This results in models that reflect the alignment preferences of OpenAI instead of the preferences of the model developers.

[gojomo]

Indeed, & an artificial uniformity of LLMs, and thought, if everyone is cribbing each others' outputs could be a concern.

But before its concern about a monoculture, the article 1st points out that mere prediction-training (on another model's outputs or fresh data) can't truly match RLHF in instilling some much-desired behaviors.

And that presents a bit of a tension with the article's 2nd concern: if mere output-mimicking *can't* match more-sophisticated training, then it can't really create the concerning uniformity, either.

And maybe: the dose makes the poison. A little cribbing might be a beneficial partial accelerator for smaller teams & newer projects, even if a lot is ineffective (at ever fully replicating OpenAI model behaviors), or deleterious (if effective and also overdone).

So the article isn't really a strong case for not trying this at all – just for keeping the potential limits & downsides in mind, in any experiments with this technique.

[laprise]

I understand what you mean, and that's a fair point. However, as John Schulman pointed out in his talk, it is possible to clone the behavior of the model, but it won't work to avoid hallucination since the underlying pretrained models are different. If we clone ChatGPT's behavior (by using its output), we'll get the worst of both worlds: weird output coming from its RLHF step AND hallucination.

[gojomo]

That seems like an experimentally-testable prediction - that attempts to "clone ChatGPT's behavior (by using its output)" will necessarily "get the worst of both worlds: weird output coming from its RLHF step AND hallucination".

The reasoning that the results won't be quite as good as RLHF, or result in a perfect 'clone' of ChatGPT's capabilities, seems pretty good to me.

But the idea it won't be helpful at all, especially to projects that are just seeking some incremental advantage? Seems speculative.

In particular, when you read the linked comments from Yoav Go, he outlines a potential RL process that uses automated scoring for non-exact similarity to preferred answers. Using known (or even 'probably') good answers from ChatGPT output, as the inputs to that process, seems like it could often offer some of the same sort of improvement to other models as ChatGPT obtained via its RLHF.