Ask HN: Small Language Models instead of LLM. Does it makes sense?

[alexei_rudak]

Everyone is talking about Large Language Models like ChatGPT, LLAma, BLOOM etc that can do any task with text transformation. I did many tests of them for changing style, locale, grammar correction etc and saw that quality is quite average. The performance is so slow and unstable to use them in scaling business.

I tried to make several small language models using 3-5 million of parallel sentence dataset to make specific linguistic transformation and see much potential there. The difference in performance between ChatGPT4 and my small models is almost 1000 times.

Anyone do such things ?

[omeze]

I think it depends on the use case. The big win IMO is cost savings for developers, tiny models can run on devices so theres no need for inference servers. The real challenge is coming up with a use case that a tiny model can perform. I suspect there’s probably a lot though, like rephrasing or gmail-like suggestions. Theyre tiny features, not full apps, but still seem valuable to me

Fyi me and a friend made a demo app of tinystories we just submitted: https://news.ycombinator.com/item?id=36960333 - from testing it out, anything with <500ms latency and 15 tok/sec feels real-time, and that makes a world of difference in UX

[alexei_rudak]

I mean the case when tiny models can outperform LLM on certain tasks. For example storytelling, and only for Harry Potter world. Something very niche but with maximum quality that can't do other tasks. Just want to see an example of that tiny model.

Did you try to optimise your model for specific tale, game characters etc ?