[simonw]

I ran a survey on Twitter over the past few days asking for successful case studies that produced economically valuable results from fine-tuning LLMs.

I ask a version of this every six months or so, and usually the results are quite disappointing.

This time I had more credible replies than I have had in the past.

Here's my thread with highlights: https://twitter.com/simonw/status/1979254349235925084

And in a thread viewer for people who aren't signed into Twitter: https://twitter-thread.com/t/1979254349235925084

Some of the most impressive:

Datadog got <500ms latency for their language natural querying feature, https://twitter.com/_brimtown/status/1979669362232463704 and https://docs.datadoghq.com/logs/explorer/search/

Vercel run custom fine-tuned models on v0 for Next.js generation: https://vercel.com/blog/v0-composite-model-family

Shopify have a fine-tuned vision LLM for analyzing product photos: https://shopify.engineering/leveraging-multimodal-llms

[donkeyboy]

Finetuning is pretty much necessary for regression tasks. Also useful for classification since you can get the direct probabilities in case you want to do some thresholding.

[daxfohl]

I imagine it's pretty bad risk to reward ratio for most companies. Especially when just tossing some stuff into your system prompt is an option.

[simonw]

Yeah, that's my assumption too. Fine-tuning is really expensive in terms of skills and time needed to attempt it, and there's a very real chance that your attempts will fail to make a meaningful improvement over being smarter with your prompts.

Even worse, even if you DO get an improvement you are likely to find that it was a waste of time in a month or two when then next upgraded version of the underlying models are released.

The places it makes sense from what I can tell are mainly when you are running so many prompts that the cost saving by running a smaller, cheaper model can outweigh the labor and infrastructure costs involved in getting it to work. If your token spend isn't tens (probably hundreds) of thousands of dollars you're unlikely to save money like this.

If it's not about cost saving, the other reasons are latency and being able to achieve something that the root model just couldn't do.

Datadog reported a latency improvement, because fine-tuning let them run a much smaller (and hence faster) model. That's a credible reason if you are building high value features that a human being is waiting on, like live-typing features.

The most likely cases I've heard of for getting the model to do something it just couldn't do before mainly involve vision LLMs, which makes sense to me - training a model to be able to classify images that weren't in the training set might make more sense than stuffing more example images into the prompt (though models like Gemini will accept dozens of not hundreds of comparable images in the prompt, which can then benefit from prompt caching).

The last category is actually teaching it a new skill. The best example here are low-resource programming languages - Jane Street and OCaml or Morgan Stanley and Q for example.

Jane Street OCaml: https://www.youtube.com/watch?v=0ML7ZLMdcl4

Morgan Stanley Q: https://huggingface.co/morganstanley/qqWen-1.5B-SFT

[daxfohl]

Have you heard of any attempts to bake MCP definitions into LoRA adapters? I've been wondering if that's a viable approach, so you don't have to put them all in context, and toggling them on and off would just be a matter of applying or unapplying the weights. That seems like it'd be more robust than putting "enable FooMCP" "disable FooMCP" etc in the context, which I'd think would trip up the LLM eventually. And it would avoid full rebuild of the KV cache that'd be required if you fully removed FooMCP from the context prefix.

Depending on use case you could either insert the LoRA weights as their own layers at runtime (no time to create, but extra layer to compute each token), merge them with existing layers (initial delay to merge layers, but no runtime penalty after), or have pre-merged models for common cases (no perf penalty but have to reserve more storage).

[simonw]

I've not heard of anyone trying that, but I don't think I've been looking in the right kinds of places.

My current mental model of LoRA is that this would be unlikely to Work, but I've never used them so I don't really know what I'm talking about. Would be a very interesting experiment!

[CaptainOfCoit]

If people have ideas for use cases where fine-tuning can make a big difference, but don't have the time/resources to try it out yourself yet want to see if it'll work, feel free to share your ideas as I'm currently creating a bunch of examples of this and could use some inspiration, I only have 3 real/confirmed use cases as of right now.

[coredog64]

Something that's in my personal backlog is fine-tuning of TrOCR for purse seine observer workbooks. The default TrOCR is expecting English words, and so the FAO species codes used in the workbook result in terrible accuracy. LLMs do poorly in this space because you'll commonly see repeats (e.g. 100 out of 120 samples all have the same species code) which then leads to hallucination.

[CaptainOfCoit]

You might enjoy this :) https://news.ycombinator.com/item?id=45640594 (DeepSeek OCR)

[leobg]

Many ppl think to fine tune an LLM on domain knowledge means to feed it chunked text of, say, psychology books. That is, of course, a wrong application if your goal is for the model to become an expert psychologist. You want the behavior of applying psychology, but you are training the behavior to write about it. TL;DR, many fine tuning fails are due to wrong dataset curation. On the orher hand, if yiu get the dataset right, you can get a 7B model outperform a 180B one.

[ACCount37]

Transfer learning is a thing. But the issue with the gap is that the datasets for "applying X" aren't easy to come by.

[ijk]

There is an awful lot of "looking for my keys under the street light" going around these days. I've seen a bunch of projects proposed that are either based on existing data (but have no useful application of that data) or have a specific application (but lack the data and evaluation required to perform that task). It doesn't matter how good your data is if no one has any use for things like it, and it doesn't matter how neat your application would be if the data doesn't match.

I'm including things like RL metrics as data here, for lack of a better umbrella term, though the number of proposed projects that I've seen that decided that ongoing evaluation of actual effectiveness was a distraction from the more important task of having expensive engineers make expensive servers into expensive heatsinks is maddening.

[ACCount37]

The importance of having good metrics cannot be overstated.

On the "applying X" problem - this almost feels to me like another argument against fine tuning? Because it seems like Applying can be a surprisingly broad skill, and frontier lab AIs are getting good at Applying in a broad fashion.