[steno132]

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them.

I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.

[adam_arthur]

The entire universe of automation projects that can be run effectively for free relative to SoTA models?

I don't think many realize that most LLM embedded automation, pipelines, products will soon be able to run extremely cheaply on models < 100B parameters.

Frontier models will be used for coding/creation use cases, yes. But for all the pseudo-deterministic, pipeline, analysis style things there will be no practical benefit to running frontier models, only additional cost.

Gemma 4 26B outperforms most 100-200B models that I've tested for reasoning and structured output.

Gemma 4 12B can consistently select where to click on browser images given a minimal prompt, and do so very quickly.

[dofm]

The 26B model is really surprising, and it is impressively concise — it spends a lot less time dithering than Qwen3.6.

[steno132]

Practically if you're running a small personal automation project you're not going to want to waste a lot of time configuring and tuning a local model. You want to build the automation and move on.

If you're building a automation as a company you definitely won't want to take on the long term maintenance overhead of running your own models for some automation project.

[adam_arthur]

These small models exist in the cloud and are/will be priced commensurately to their size.

Your claim is effectively that companies don't care about operational/cloud costs. Even pre-LLM, companies regularly assessed and tried to pare down cloud spend.

[sowbug]

Whatever you're doing, try doing 500 or 1,000 of it in a batch. You'll exhaust any subscription quota you have, or if you're paying per token, you will probably find it too expensive. That's when you'll start to ask "how smart a model do I really need for this job?", and you'll investigate running a small but sufficiently capable model on your own PC, churning overnight through your 1,000 tasks.

[mikeocool]

> I've been using Claude and GPT models for years

All 3 years?

[steno132]

GPT1 was released in 2018, so yes, since then.

[victorbjorklund]

GPT1 was way worse than small Gemma’s are now.

[Zambyte]

I like using my computer.

[steno132]

Exactly, thank you, we are on the same page! It's great to be able to use our own devices and not have their compute coopted by a third party.

I'd rather not have intensive compute needed shifted onto my personal machine which I want to use for something else.

[satvikpendem]

By that logic, any software you run that isn't fully built by yourself is "third party" therefore you shouldn't run anything at all on your machine, thus obviating the need for it entirely.

[steno132]

But practically AI inference requires substantial local computing resources. It's not some web app, it's a order of magnitude more compute needed

[Zambyte]

Hopefully now you understand why people want smaller models.

[satvikpendem]

Not really, I run a production service on a basic server using these Gemma models, the server is weaker than my MacBook. Most people's laptops and even phones actually can run local models, most simply don't know how. Run Unsloth Studio and you'll see how easy it is.

As the sibling says this is why people want smaller but still performant models.

[Zambyte]

I am not a "third party" on my own computer.

[user2722]

There is tinfoil.sh as well but honestly running this stuff on an airgapped server allows a better peace of mind about the data being used for something else.

[steno132]

What's wrong with the data being used for something else? Someone is providing digital intelligence to us, saving us many hours a week, so the least we can do is provide them a little data so they are able to improve their service.

It would be selfish and unethical not to in my view. And ultimately the data is just being used in order to improve the models and benefit us, not for anything nefarious.

[NicuCalcea]

If sharing our data is the least we can do, they shouldn't also ask us for our money. Otherwise, it's more than the least.

[mannanj]

I don't like the gaslighting of paying Anthropic or Open(Closed)AI and it being said its unsustainable for them to take my payment while simultaneously they take my data (edit: which is incredibly valuable) and I cannot opt out of that.

The obsession is for leaving hostile and abusive entities, the corporations or the people who fund them that have a horrible track record in regards to ethicality, rights and respect & human dignity.

[steno132]

My view is, if you're going to use the service - you should give the data.

It's like using Gmail and expecting them not to train their AI models on your data - how can you expect that when they're giving you a secure, reliable, highly functional email client completely for free?

The digital economy only works if everyone pays their fair share. If you don't want to give your data then you are really harming everyone by slowing down AI development for everyone else.

[klardotsh]

Because we pay for the models.

If I pay you for a service, what implicit right should you have to then continue to profit in perpetuity by storing the data I paid you to process?

If LLMs were free your Gmail analogy might hold up. They aren’t, and so it doesn’t.

AI development can continue with the data folks opt into, or with the data AI companies incessantly scrape with reckless disregard for polite system loads. AI development does not require retaining all user inputs forever.

[mannanj]

However, you didn't actually get what I meant down, so you ended up inadvertently Straw Manning me.

My disinterest is in sharing my intellectual IP. Most people up to now, have never shared this much of their intellectual IP with a company. Name one product through human history before that got this much data and insight into human thinking and now can use your most intimate conversations, ideas and needs for non-training purposes?

You can't even opt out of that! At least for the training data you can opt-out.

[satvikpendem]

Intellectual "property" is not real property. While I disagree with the parent on many things as my comments show, IP is not one of them. Information should be free, for anyone and everyone.

[mannanj]

Another straw man.

"real" property or not. You agree that we have some right to our own outputs, right? Is that not dignity, to say "I want my outputs protected".

Seems like you think that your ideas should be free, as you called it information. How about you back that up with action... please send me all your most intimate, valuable ideas. Oh no, you don't feel comfortable? Then why are you sharing it with companies?

[satvikpendem]

Sure, it's not like ideas are protected anyway even in the current copyright regime. And anyway I share it with companies because they perform some useful service for me, such as chatting with LLMs about said ideas.

[mannanj]

Apple is a good example of ethical services. They still give you privacy and ownership of your data, you keep your dignity and data. Google is a horrible model for this - it matches the whole thing about unethical, abusive, gaslighting relationships I described.

[satvikpendem]

The same Apple that takes 30% of all transactions? In reality no corporation is in the public interest.

[mannanj]

straw man. I'm talking about data.