[ux266478]

As parent implies, they're testing the wrong control mechanism. Why are you using policies instead of real controls over the weights and inference pipeline?

Well the answer is that VC-backed companies decided AI is not a domain expert tool for highly competent technical users, it's a magic oracle for the lowest common denominator. So you don't get any of the actually useful controls, just context engineering like that's fucking sane at all. It's like trying to program by navigating a git history. Not writing any new code, you don't have the ability to do that. No, you exclusively have the ability to move around a git history and cherry pick things. It's insulting that they want to charge money for this shit.

[giancarlostoro]

> just context engineering like that's fucking sane at all.

Meanwhile every model is at a battle to figure out how to stop you from jailbreaking it, which I assume takes a toll on your ability to prompt. Heck, if you read r/Claude and related subs during the early release of Fable there were a lot of complaints that it would downgrade you heavily.

[Aurornis]

> Well the answer is that VC-backed companies

Look, I enjoy local LLMs as much as anyone, but I think there is some motivated reasoning happening in this thread to try to make local LLMs sound like a utopia against those evil VCs.

Local LLMs suffer from the same problems.

[ux266478]

And I think you didn't understand what I wrote, so let me reiterate: Of course policies are equally ineffective at strictly governing the behavior of local models. Why would that change? The actual problem is that somebody is attempting to misuse them for that in the first place. There are only two reasons for it:

- They have no other options

OR

- They have no idea what they're doing

Local models solve the first problem, and let you apply the appropriate tool by virtue of the control you have. The alternative does not.

In-context, your reply reads like you're asserting that anthropic's offerings give you all the same control as a local model, and that having this baseline of control over a computer program is "utopian". To me that sounds more like the second problem, and nothing can help you there.

[Aurornis]

> In-context, your reply reads like you're asserting that anthropic's offerings give you all the same control as a local model, and that having this baseline of control over a computer program is "utopian".

No, I’m talking about the idea up thread that having control over more settings removes all LLM defects with local models.

I have all the control I want over the local models they run. No combination of settings is going to make them outperform Anthropic’s models.

[ux266478]

He didn't say that, and I didn't say that, because in fact neither of us were talking about just settings, so that's a pretty curious idea to be addressing.

> No combination of settings is going to make them outperform Anthropic’s models.

And even this is addressing something nobody was saying, and manages to come off as a marketing read.

[throw10920]

> In-context, your reply reads like you're asserting that anthropic's offerings give you all the same control as a local model

It absolutely does not.

[TiccyRobby]

And what is the difference between cloud modeld vs local inference regarding this? It is still unclear in this thread - asking out of curiosity.

[ux266478]

The ultimate difference isn't necessarily down to remote vs local. You can run a "local" model on the cloud the same way you could on a local machine and have all the same control over it. You can modify the weights, inject behavior, stack on arbitrary LoRAs, use whatever runtime you want, etc.

The problem arises in the case of anthropic, openai, etc. where you can't do those things. I think they have LoRA services that they'll charge you for, but that's about the extent of it. Think of it like having a compiler and its source code, and the operating system it runs on vs simply having access to invoking it via a web service.

[satvikpendem]

Changing the weights and controls does not alleviate the context limits causing LLMs, local or cloud, to forget over time. I was asking how local LLMs would amelioratile this but looks like there is no solution.