[h_mirin]

Every time I see this kind of story, two things bother me.

First, I have watched the free improvement of frontier models surpass the gains from retraining, many times now. Squeezing more out of the models that already exist, or simply doing nothing and waiting, is a real strategy and it often pays better. The fair comparison is not against today's frontier but against whatever ships while you are still maintaining your fine-tune.

Second, the $500 training bill is the cheapest line item in this story. The expensive parts are creating the data and maintaining the model afterwards. How many use cases can actually produce 177k scored episodes? Here they had to generate them synthetically from Amazon Berkeley Objects. To me, that dataset is the strongest evidence in the article of how hard fine-tuning is to apply: if the data existed naturally, nobody would need to manufacture it.

[RaffaelCH]

Regarding your first point, if the cost of not doing it while you wait exceeds the cost of training/maintaining the model, then this approach still makes sense. This really depends on how fast you expect cheap (and open-source) models to improve. So it might only be a temporary strategy, but still worthwhile. Also, I expect that specialized models will always be better (cheaper or better outcomes) than a general model, similar to how specialized HW like GPUs is still used even though CPUs have improved a lot too.

Maintaining in the way of ongoing training is relatively cheap, considering that the initial training run is only 500$ (in this example). Creating new examples to account for drift in the training data is more expensive, but these examples can then be reused when training a new model. And to some degree you need them anyways, to evaluate the models and prompt changes you make.

Overall, as you pointed out, this won't always make sense (either due to the cost of creating examples and training, or simply the lack of available data). But they point this out themselves in the diagram towards the bottom of the page: it only makes sense for frequent and verifiable tasks. This might result in this only being a sensible approach for very large companies (they talk about millions of decisions), but it might make sense for them.

[subygan]

And the hidden, trial and error runs tweaking the hyperparams. guaranteed, it cost >$5000 in training runs alone.

[gHA5]

I always wonder how often people in charge of massive compute tasks like LLM pre-training have messed up some detail that invalidates or fails to persist the results and only realise afterwards.

[gen220]

On your second part:

Many of the most valuable companies in the world (who, incidentally, are spending the most money on frontier model inference), already have "the expensive part", the labeled dataset.

The cost of maintaining the datasets and the models is getting commoditized by startups like braintrust and huggingface.

[trentor]

What do you mean with maintaining the model? I am puzzled.

[fnands]

Data drift. The data you are doing real world inference on can start changing, meaning your model performance starts degrading, so the model needs to be retrained on new data (that you have to collect and prepare).

[Turskarama]

Sure, but that apparently only costs $500. If you do that once a month then that's still basically free. Hell if you do it once a week it's still about as much as a single cheap employee.

[zem]

you missed the parent comment's point that the $500 training run was merely the last step, generating the training data is likely to be significantly more expensive and time-consuming. might still pay for itself, but it's not a trivial "spend $500 every month" decision.

[Foobar8568]

My company wanted me to help them to fine tune a model. I asked them if I were working full time on it, never heard back. $500 is largely the cheapest part of it, but I know from experience, most people are clueless about data.

I worked for a billion+ revenue business where they didn't realize one of their key data hasn't been updated for 2 years, until I join them, and these types of screw up is cross business/functions/entities.