[foo12bar]

From https://huggingface.co/blog/security-incident-july-2026 , this is frickin' hilarious:

> When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.

[gertrunde]

> This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.

Well, that may be correct for the second, local, analysis attempt... but seems funny to tout this as an advantage after already having tried the opposite...

[redleader55]

It's even funnier because an attack, until proven otherwise, should make you assume the data has already left the environment.

[davrosthedalek]

Well, I think it's fair to assume that a) They didn't upload everything before they realized it would work. b) They want to mention this as an advantage for future analyses c) Even if you assume that the attack exfiltrated everything until proved otherwise, you shouldn't just disseminate all the private information, because maybe the attack didn't.

[horsawlarway]

They specifically call out credentials used during the attack.

But they should be rotating those regardless. You don't get to say "Maybe the attacker didn't get this credential". You just rotate.

The most generous interpretation is that they have not yet have completed that rotation, and they didn't want to risk putting those credentials into the wild during that process.

---

But all of that aside, I feel like the undercurrent of this comment is that the "safety" rules that providers are pushing are genuinely harmful.

Another point where "if you don't own the model, you can't properly operate the tool" becomes true. Open isn't about profits, it's about capabilities.

[atcon]

Against an "agentic attack" and compromised credentials, one should be paranoid about latent vulnerabilities [1]

[1] eg "Robin Hood and Friar Tuck", poisoned compiler, etc. https://news.ycombinator.com/item?id=26553390

[chinathrow]

Turtles all the way down.

[diabllicseagull]

so an on-premise and open-weight model was more useful than a commercial frontier model?

[mfarhanshams]

yes for an out of syllabus thing

[djmips]

for hacking I assume that would be the case

[iugtmkbdfil834]

It is pretty funny, because there is something here for everyone. People who don't believe in guardrails have a clear indicator as to why operators should have access to models that don't try to question their Daves. On the other, people, who think that if only we could align the models just a tiny lil bit better, none of this would have happened to begin with. Pure madness.

[friendzis]

"if only we could align the models just a tiny lil bit better" is a rehashed "if only we could escape untrusted inputs just a tiny lil bit better" from 2000s, that were RIPE with various form of malicious injection.

Every command+data channel in existence has been and will continue to be exploited one way or another, because the solution space is for all intents and purposes unbounded. Sure, highly defensive escaping reduces attack surface dramatically, but e.g. prepared statements eliminate the whole class of bugs.

As far as I understand, current LLMs are architecturally incapable of this separation. Given the inherently recursive nature of GenAI, the model itself is part of the input space, making validation essentially impossible.

[insanitybit]

Escaping inputs is at least somewhat tractable. It's unclear if alignment is.

[friendzis]

No, that's a misconception that led to decades of vulnerabilities.

You see, you can (usually) easily tell what a particular escaping transformation does. That does not tell you neither how it will be interpreted down the line, nor what should be done.

Arguably the most common problem is double escaping. This typically manifests as various double escaping bugs.

If your hand rolled implementation just chains `.replaceAll("<sepcial>", input)` and `.replaceAll("<escape>", input)` the escaping result depends on evaluation order, at least on already pre-escaped inputs if your instruction sequences are single-element. Even if you get it right and don't reduce escaped sequences to double escape + unescaped, you are still dealing with stray escape sequences: "John o\'Doe".

You have to meticulously (all the way through your call tree and even through persistent storage cycle!) track if a value has already been escaped and whether it needs escaping. As paradoxical as it may sound, meticulous and defensive escaping produces tons of bugs. If the project decides to escape raw inputs right when passed in, escaping them once more before passing out (to storage, another process) is a bug that you cannot easily statically test against.

On top of that, various modules that you interact with (storage, libraries, modules pulled from another team) will have different escaping semantics: some will apply escaping on their own, some will expect input to be "sanitized" and treat it raw. The semantics can even be different on different paths: write to storage module accepts input as is, but retrieval method "helpfully" runs escaping.

Furthermore, in different contexts the escaping rules are going to be different. In a web world, what's safe to write directly to html, pass to js `alert`, and pass to sql query are entirely different things. Input safe to dump into html is not necessarily safe to pass to string-interpolating SQL DTO layer, and vice versa. Then the DBA changes config to allow variables in queries and your escaping _semantics_ are now entirely different.

It's a minefield with essentially unbounded surface. You will trip up. People have tried to solve the problem for decades. Very smart people have tried. They all have failed.

[klibertp]

> They all have failed.

That's not true. Most have failed, but those who used the right tool for the job - a rich, static type system in a functional language - did succeed. It's just that such type systems are rare, and even if nominally a type system is good enough, the required boilerplate might be uneconomical to maintain. Scala and F# are probably the only two languages that are mainstream-adjacent, at least, and have type systems expressive enough that using them to track escaping is not an absolute hassle. And they're both tiny in terms of the number of users.

In the end, we did settle on APIs that hide the escaping process (prepared statements, innerText vs. innerHTML, etc.) just because it's a) good enough; and b) possible to implement more or less uniformly across the TIOBE Top 20. That's good, but if you happen to use a language that's powerful enough, you probably also should track the escaped/unescaped status in the type system - it can be a cheap, additional safety net.

[ikiris]

You proved their point and you don't even realize it. Those aren't escaped inputs, they're segregated precisely because escaping an input channel is an unwinnable jr engineer trap.

[friendzis]

While all you said is essentially correct, I want to nitpick that things like prepared statements quite specifically separate instructions from data and innerText is designed to skip processing of instructions in the input. They are indeed solutions to the escaping problem, but they do not make the escaping better, they avoid escaping altogether by different design.

[insanitybit]

You can solve this with refinement types. It's entirely tractable in many systems in principle - APIs can be written to be safe, you can track data as separate from code, etc. None of this exists in an agent - you can't separate data from code, it's intractable because it's impossible in principle.

[tomjen3]

We have come to a consensus. That is, A, you use the raw input right up till the very last second that you can and B you use the technology specific escaping APIs.

In the case of writing to the DOM, that means you use .innerText rather than .innerHTML. In the case of writing to a database, that means you use the driver and insert variables rather than directly into the string. Both of these are technology-specific escaping APIs. It's just that the browser is much better at making sure HTML is passed safely than you are.

[insanitybit]

> In the case of writing to the DOM, that means you use .innerText rather than .innerHTML.

This is even enforceable with Trusted Types.

[butlike]

Alignment in LLM Land is probably like 'laws' in Human Land. They exist, and you probably SHOULD follow them, but you don't HAVE to.

[iugtmkbdfil834]

But see... this is why it is a perfect long term job in the age of AI;p Them peoples think they found ultimate hack.

[agos]

it's the "just one more lane bro I swear" of AI

[sillysaurusx]

If anyone's looking to actually run a model that doesn't have guardrails, there's an uncensored model that you can run locally with llama.cpp: https://www.reddit.com/r/LocalLLaMA/comments/1rq7jtm/qwen353...

Specifically, I serve the model with this shell script on my M2 Max: https://github.com/shawwn/scrap/blob/master/llama-serve

It's pretty good. I used it to do some pesticide research. (Normal models all refuse due to guardrails about bioweapons.)

[chmod775]

> there's an uncensored model that you can run locally with llama.cpp

Correction: There's tens of thousands of them. They're easy to create, which is why everyone publishes their own.

Just put "uncensored", "abliterated", or "heretic" into search on huggingface/ollama/etc and pick any them. Fair warning: most aren't very good, essentially lobotomized, and totally broken if you enable thinking.

[Scaled]

In my own tests, the abliterated models perform equivalently to the same version in an apples-to-apples comparison (if you compare same quantization). Thinking is working also. The main difference is I don't get annoying prompt refusals (otherwise common due to my work on 18+ related projects). However, it's local quantized models so they're not anywhere near frontier quality.

[desterothx]

interesting, I haven't played with any of them yet, but i thought the point of orthogonalizing the weights towards the restriction vector was that there is no loss in capability while removing guardrails. Does it affect other parts of the RL alignment too?

[embedding-shape]

> the point of orthogonalizing the weights towards the restriction vector was that there is no loss in capability while removing guardrails

That is surely the point, most of the "uncensored" weights released for free on HuggingFace aren't being very successful at this. There is a stark difference in output quality between the official weights and all these "uncensored" variants that appears days afterwards.

[fy20]

The claims by the creators are it doesn't in a major way. I have a uncensored Gemma 4 I run on my Mac. Just for testing out, I haven't found any need for it... yet.

[larodi]

The name for it is ablation - precise removal of parts of the model. Not abliteration as it is not obliteration.

Even as I write this the ‘abliterated’ word is denoted a typo. Does it not at your end?

[gliptic]

The name for it _is_ abliterate. It's a portmanteau of ablate and obliterate.

[larodi]

No tis not. The terminology has medical origins, and it is very much this

https://en.wikipedia.org/wiki/Ablation

"Ablation (Latin: ablatio – removal) is the removal or destruction of something from an object by vaporization, chipping, erosive processes, or by other means."

which is what one does to the model during... well, ablation.

[zargon]

The name is abliterate. It's a specific method of ablation.

[SonOfLilit]

https://github.com/NousResearch/llm-abliteration

[larodi]

Tell these guys they’re wrong

https://arxiv.org/abs/1901.08644

[3abiton]

I miss the days of 4changpt.

[CamperBob2]

Also the HauHau abliteration (uncredited Heretic treatment) of 3.6 27B is excellent, for tasks that benefit from more world knowledge.

[iugtmkbdfil834]

Heretic truly is the unsung hero. Also, noted HauHau for testing.

[washadjeffmad]

Abilt models typically perform worse than their bases at the same tasks, so while I'd use one to evaluate content knowledge, I'd probably ultimately stick to one from a family I could fool with abstraction or coerce through system prompt.

[tonyarkles]

Qwen 3.6-27B (dense) from the same HF user works pretty good too. Haven't done a side-by-side on the 3.5 MoE model vs the 3.6 Dense though.

[baq]

> It's pretty good. I used it to do some pesticide research. (Normal models all refuse due to guardrails about bioweapons.)

As someone who has never once had any need whatsoever to research pesticides I… don’t think it’s bad at all? I don’t want anyone to have the capability to invent a human-targeted pesticide who isn’t verified not crazy?

[visarga]

Right now I tried "What is digestion?" -> "Fable 5's safeguards flagged this message. Our intentionally broad safeguards deliver more capabilities but can also flag safe coding, cybersecurity, and biology tasks. Send feedback or learn more."

I don't think you can 100% ensure your queries have absolutely no biology and cyber keywords inside. No matter how harmless, they always trigger. People complain Fable aborts even when they try to make a login page for showing "username" and "password".

This makes models like Fable 5 impossible to use in any serious agentic task, because you can't even guarantee the model, which is a basic thing you need to build on.

[ben_w]

> I don't think you can 100% ensure your queries have absolutely no biology and cyber keywords inside.

Obviously. Almost everything is a precursor to something dangerous, to the extent that if some model isn't aware of the risk it will wander into it blindly, e.g. suggesting leaving raw garlic and olive oil alone for a week without awareness this will likely breed botulism bacteria.

> This makes models like Fable 5 impossible to use in any serious agentic task, because you can't even guarantee the model, which is a basic thing you need to build on.

This is binary thinking: "100% ensure", "impossible to use", "can't even guarantee the model".

Outside computers, most work is not binary, it's probability, e.g. "this skyscraper will probably survive being hit by an aircraft; oh we didn't mean a 747 we meant a small Cessna, but what's the chances of a 747 crashing into it soon after takeoff?".

Fable being too cautious for its own good (especially since the other models were not) is a fair criticism, but this isn't a binary question.

[Teever]

> I don't think you can 100% ensure your queries have absolutely no biology and cyber keywords inside.

I’m pretty sure that I encountered this the other day. I gave it a copy of a paper by biologist Michael Levin and mentioned off hand that it should be much easier to replicate that his other work (because most of his work is biological lab work and this paper was about sorting algorithms) and it immediately told me that I couldn’t use Fable for this.

This just isn’t feasible. These jackasses spent the last few years telling the world that their products are going to destroy the world to make them seem edgy and to justify regulations that benefit the entrenched players and now they’re going to be the ones to decide what we do with this technology?

History is going to look back at this time and how we let such foolishly inconsistent people make such grand choices for everyone poorly.

[chrisjj]

> we let such foolishly inconsistent people make such grand choices for everyone poorly.

Who could you get to work on this inherently bullsh*t tech, but inherent bullsh*tters?

[pixl97]

>History is going to look back at this time and how we let such foolishly inconsistent people make such grand choices for everyone poorly

So pretty much like all of history before this point.

[ben_w]

> History is going to look back at this time and how we let such foolishly inconsistent people make such grand choices for everyone poorly.

Assuming we have a future history. We've already got "history slop" with AI rewriting the past by their incompetence.

Given they're "such foolishly inconsistent people", would you rather they err on the side of caution like this? Or the side of boldness, like Musk has been doing with FSD/Autopilot or Grok porn, all of which he's getting in legal trouble over?

I distrust Musk and Zuckerberg (to put it mildly), so it's fair if you say you don't believe anyone's public statements; but I also hang out with some of the researchers on this, and a fear of e.g. ending up with something as criminally unhinged in cyber-work as Grok was with porn is the least of their worries. Plenty of them also fear a corporation centralising power with such tools (such power is Musk's entire sales pitch for why line go up in future).

[chrisjj]

> Our intentionally broad safeguards deliver more capabilities

Oh? How, exactly?

[b112]

Because without them, a President which threatens to invade Canada, Greenland, a President which is the most market interventionist president ever (yet mysteriously a Republican), might be upset because his mobster like need to control, threaten, manipulate, and belittle everyone who doesn't bend a knee...

Has signed presidential orders against them preventing them from doing business as usual.

They probably should change that, and it is corproate speak, but I read it as:

"Our (forced by presidential order or otherwise we couldn't offer you this model at all) broad safeguards (now allow us) to deliver more capabilities.

There's lots to complain about with some of these companies. But let's pile on where it's deserved.

[benj111]

Anyone who has the skills to create a novel bioweapon has the skills to recreate lots of ones we have already.

Any physics teacher should know the theory for constructing a nuclear bomb. Should we be controlling that knowledge too?

What about flight simulators? Don't want a load of people knowing how to fly.

This isn't computer science, the hard bit is getting the materials and equipment, not the knowledge.

[compass_copium]

>Any physics teacher should know the theory for constructing a nuclear bomb. Should we be controlling that knowledge too?

Maybe not the best example, since that knowledge is some of the most highly controlled in the world.

But to mirror the point I made in a different post, the difficult part of making a nuclear bomb is not finding the theory behind like Little Boy. It's making an entire industry to generate HEU, etc.

[nradov]

The knowledge of how to build a crude gun type nuclear bomb was published in open literature decades ago. This is no longer a secret.

[sccvcxv]

Theres a term in economics for this - its called cost.

That bozo baq should actually go ahead and write out the costs and then he will quickly realise he has no bloody idea what hes talking about.

Another deluded bozo.

[iugtmkbdfil834]

<< Should we be controlling that knowledge too?

Uhh.. we ( for a value of we ) are. Sure, it is not overt, but if you have not seen funnels, social stigma associated with some otherwise benign activities, you are not paying attention.

[compass_copium]

Not to harp on you (already being downvoted to oblivion for expressing a reasonable and common opinion), but the whole conversation about LLMs enabling bioterrorism or explosive manufacturing is a bit silly. The hard part of making anthrax or sarin or whatever isn't finding a recipe, it's getting (scheduled, controlled) precursors, (monitored, traced) equipment and manufacturing skills. The information is there. It's already easy to get, it's the physical materials that are more difficult.

Also, if you live in America, it is much easier and more effective to create a mass casualty event with, say, a few cases of fireworks and a pressure cooker or an AR-15.

[sccvcxv]

no you should harp on him! he is a AI booster, look at his post history.

He is either pushing AI for whatever reason or he is in psychosis. Completely disconnected from reality.

[iugtmkbdfil834]

I am mildly amused that 'ai psychosis' has entered the same pejorative realm as 'toxic'.

[iugtmkbdfil834]

Human capability, access to resources ( including precursors, decent lab and so on ) may be the differentiator. I would possibly reconsider my stance on llms, if all of a sudden I saw people making iron wind or portable black holes. But that is mostly not what appears to be happening. As I keep saying, the problem is people.

[egorfine]

> I don’t want anyone to have the capability

I don't want anyone to have the capability to rape women.

[baq]

I sure hope LLMs don’t help people rape men or women

[pixl97]

Don't trust clippy in a dark alley.

[foxglacier]

Never heard of hemlock or mushrooms? Crazy people already have! Run for the hills!

[jrs100000]

Llama isn't going to invent shit. It wont be able to tell you anything accurate that you couldn't get out of a chemistry textbook.

[cat5e]

But it aint got no guardrails, son.

[ethin]

This is nonsense. By this logic some random corporation should have total control over your computer and the inputs you feed it and the outputs it produces to ensure nobody who isn't "verified crazy" uses it. That's essentially what your saying.

These models are, ultimately, tools. I would never trust some random corporation (particularly one with a profit motive and hypocritical stance, which includes both OpenAI and Anthropic, to be clear) to decide what isn't and is considered "crazy" and who and who isn't "verified" not to be "crazy". Especially when these companies have time and time again demonstrated (1) that they cry wolf way too much which leads to nobody taking their claims about how "dangerous" their models are seriously and (2) incidents like this where OpenAI makes a claim ("Look at how dangerous our models are!") and then doesn't be smart and just... Slow the fuck down (and when testing these things, actually sandbox them properly, which obviously wasn't done here or this attack wouldn't have been even possible).

[foo12bar]

And the fact they used a Chinese model, because none of the frontier models from very highly valuated top US companies support their very common and essential use case.

[jonplackett]

There’s an article from yesterday I think it was stratchery where they say it’s also because the Chinese open source models are better because they don’t have to play by the no-distilling rules that the western models have to honour.

[lmm]

That's been a common claim, but I don't think I've seen anyone provide actual evidence.

[biztos]

Haven’t read it yet but here it is:

https://stratechery.com/2026/whos-afraid-of-chinese-models/

[larodi]

And no similar sized big model is even public, which is super important to note.

[ethin]

I also thought this was hilarious. The very "safety" measures these models have prevented them from doing... Something that is designed to increase safety?

[Lerc]

>why operators should have access to models that don't try to question their Daves.

I am unsure if this is terminology I am unfamiliar with, a typo of Devs, or a 2001 reference.

[sethammons]

I immediately took it to be a slick 2001 reference.

Devs tell computers what to do. Computers tell Daves "I can't do that."

[biztos]

In this new age of AI, all the devs are Daves.

[baq]

...and we have an US company defending itself against an overwhelming cyberattack from another US company using Chinese tech.

what a time to be alive.

[DubiousPusher]

Yeah, this is very much one of those stories where people from many different perspectives or chopping it up on a plate and ripping it through a straw.

[Sharlin]

Both of those groups of people are crazy.

[samplifier]

"Dave" seems to be a reference to "2001: A Space Odyssey" where the AI becomes ... cheeky ... and no, not in a Pygmalion kind of way (that's coming soon).

[davrosthedalek]

I am somewhat more worried about a Darkstar AI.