[leoedin]

I got excited about that, until I actually tried to download a model and run it locally and ask it questions. A current gen local LLM which is small enough to live on disk and fit in my laptop's RAM is very prone to hallucination of facts. Which makes it kind of useless.

Ask your local model a verifiable question - for example a list of tallest buildings in Europe. I did it with Gemma on my laptop, and after the top 3 they were all fake. I just tried that again with Gemma-4 on my iphone, and it did even worse - the 3 tallest buildings in Europe are apparently the Burj Khalifa, the Torre Glories and the Shanghai Tower.

I wouldn't call that effective compression of information.

[theshrike79]

Yea, it's not an encyclopedia of facts. Language models store the FEELS of the data in vectors (or angles in Gemma4's case, it's a cool thing) not the exact string.

But what you can do with local models is give them actual data and tools to search it. Download a copy of Wikipedia locally, give the agent a way to search it and BOOM accurate information without an internet connection.

Also "small enough to live on disk" is a bit vague, especially when models get super stupid super fast when you get to the smaller size. At that point they're just basically 40k servitors that can use tools and nothing much.

[MattPalmer1086]

I don't think any LLMs are good at accurately regurgitating arbitrary facts, unless they happen to be very common in their training, and certainly not good at making novel comparisons between them.