[gojomo]

Yes, the inner link to the Yoav Go writeup is the gem here, concisely explaining the benefits of RLHF-scoring over mere prediction.

Though, as Go speculates, it's likely possible to reduce even further the HF ("human feedback") part, while still reshaping the model to have the helpful qualities.

My guess is there's a rich set of potential ways to this – automate that extra level of distinction between mere "exact token prediction" & "sufficiently valuable responses" – & OpenAI probably has a few undisclosed advances here as part of their GPT4 training/tuning.

In particular, Go's suggestion that a separately tuned LLM can do a fuzzier scoring of whether an answer is "close enough" to an idealized answer seems like the sort of promising ensemble approach that will have been an obvious next step for most LLM teams, probably being tried by many independent teams right now.

[b33j0r]

It seems pretty transparent, as I think you might be implying in part, that attempting to leap-frog without directly copying training data or model weights is a temporary optimization for “bootstrapping” teams.

I see this pretty directly in stablelm releasing both a base model, and a tuned model… which is not based on the base ;)

There is a goldrush to get training sets worth using, and if something 90% quality gets your models on the map quickly, it’s an attractive option. As they say, attention is all you need.

Training in the 7B range is a lot cheaper than I expected. Fine-tuning almost negligible—if you have clean data, which has always been the expensive part.

Most humans expect more than $0.000003 per token as compensation for _your_ dataset collection.