[phpnode]

This kind of narrative is going to bite them just like the "AI will take your job" narrative has. It feels like the frontier labs are taking a massive gamble with public perception here. I assume the goal is to paint the technology as so powerful and dangerous that only a handful of blessed US companies should be trusted to run it, in an attempt to suppress the rise of the Chinese models that are rapidly catching them.

[defgeneric]

This is where we need the hardware companies and neoclouds to start speaking up. The labs want to elevate matters from the level of civil society (basically, competing firms) to the State (enclosure), and as always, in the name of security. But other actors in the same ecosystem have strictly opposed interests here, and are equally if not more credible as far as the State is concerned. If players like Nebius, Baseten, Fireworks, etc. among many others including obviously Nvidia, Dell, AMD, and so on don't get ahead of this they will be sacrificing trillions.

[shimman]

Why are you expect companies that have been profiting off of LLM insanity to do the right thing if not legally compelled to?

[classified]

Exactly, it's about taking this stuff off the open market where anyone can judge it and there's competition, into government contracts where competence to judge the offer is scarce or absent, and they can ask much higher prices. And with this much investment at stake, any lie that sells the narrative will serve.

[defgeneric]

Yes, and given the nature of the current administration, whose actors are not inclined to see themselves as independent competing capitals among others, but rather as privileged capitals, and therefore more inclined to move towards taking an interest in the process of enclosure, ensuring it includes them, the hope seems to lie with companies at the hardware layer who have an interest in seeing the diffusion of intelligence play out freely at all levels of society.

The labs have to be told NO--the problem they're dealing with, that model outputs give the game away, and that in turn the distiller becomes the distilled, is a fundamental problem they have to figure out how to deal with without going to the State.

[reactordev]

That’s the play. That these frontier models are so powerful that they must be behind sovereign firewalls and gateways.

[MarcScott]

Didn't HF use an open weights model running on their own hardware to solve the issue though? Sort of defeats that narrative and plays into one in which frontier == bad_guys and open == good_guys

[reactordev]

HF guys, especially those under Julien Chaumond, are fantastic and will use whatever they can to address their issues. Open or closed but they are firmly on the Open side of the fence. However, their storage and model service is about as sticky as you can get so they are in a different position. They don’t have to sell capability, they sell capacity and community.

[adrian_b]

Yes, they have used GLM 5.2, which promptly did whatever they asked it to do, while their first attempts to use their enterprise access to a "SOTA" model failed due to refusals to investigate anything that is security related.

[pferdone]

Public doesnt know about Huggingface. ChatGPT (OpenAI) says it‘s dangerous. They must know.

[polycancel]

>That these frontier models are so powerful

Maybe powerful might NOT be the right word to describe them, they are just non-deterministic, there for we going to see this kind thing more and more.

[AnimalMuppet]

Non-deterministic, sure. But also they are powerful, at least powerful enough to launch a cyberattack. Until this morning, that was not a power that I thought they had outside of fiction.

And, the thing is, I don't want non-deterministic things to have that kind of power. We don't want that. We want that kind of power to not be triggered by a random number generator.

[jappgar]

I don't know why you wouldn't think they could do this already.

Without the system prompt these models can be used to do all sorts of terrible things.

That's precisely why they need to be strictly regulated by international treaties.

[solumunus]

Please buy our IPO before it crashes so we can be billionaires.

[reactordev]

Exactly, this is all so they can continue with their S1 and they can dump shares onto the hedge funds.

[Razengan]

And require your age verification, selfies, and DNA samples.

[Sol-]

Ironic that HuggingFace needed Chinese models to defend against it. But of course the spin of the leading firms will just be to point at their trusted access programs and demand that all dangerous activities, even if just defensive, happen via their APIs or be outlawed otherwise.

If that's the position they take then they really should be heavily regulated or nationalized. Cyberdefense against their own models dependent on their goodwill? Sure, but then they have to sell defense capabilities at subsidized rates with a limited margin. Would be very weird otherwise to take the world hostage with their models and then also sell the solution while demanding intrusive KYC.

[oogali]

It’s like the old firewall meme, reincarnated.

https://securityzap.com/wp-content/uploads/2015/12/layered-s...

[datakan]

Are the Chinese models actually catching up or are they just distilling the frontiers? If it's all just distilling then they'll always be behind.

[f6v]

There's no reason they can't build their own models from the first principles. They have the hardware, energy, and enough CS scientists.

[chicken-stew]

I do wonder how they source their bulk literary data. Do they have a google books, an archive.org or an anna’s archive for chinese content? What’s the Asian equivalent to Elsevier? Is there (strong) copyright on the corporate level?

[amazingamazing]

Who told you they were distilling? Why might they say this? Think. It’s like complaining that the top student only does well by going to office hours instead of mindlessly reading textbooks.

[literallyroy]

Is this a better analogy?

The top student giving paid lectures about his classes, and another student skipping class and instead studying those lectures to end up with the second highest grade?

Maybe it could be improved with the other student not even going to the same school?

[amazingamazing]

In your example the problem is what exactly?

[shimman]

They don't like competition when they have to actually compete.

[AndrewKemendo]

In capitalist economics they call this the “free rider problem”

[baq]

they're all introducing themselves as claude for one, there are more quantitive and qualitative arguments elsewhere

[dormento]

To be fair, anthropic models, when asked in chinese, also used to introduce themselves as deepseek sometimes. This is a limitation of the technology.

[amazingamazing]

All chinese models are introducing themselves as claude? Why make claims trivially disproven?

[baq]

you're technically correct and missing the point

[rdtsc]

Well that and also benchmaxxing.

[gonzo41]

Agree, they also may find themselves in a place where the government rightly says they can't have their dangerous new toy because they can't be safe with it. This is a high stakes PR game. Governments can and will step in and embargo and regulate these systems in ways which will hurt the companies and investors.

[Spooky23]

Data is the new oil, AI labs are the new steel mills.

[therealpygon]

1000% the case. Admitting they failed at security and allowed privileged escalations in a prompted AI would look bad for them; the AI just did it all itself is FUD to boost the arguments for regulation. And most news won’t challenge this FUD because it gets clicks.