[6gvONxR4sf7o]

The framing they use is hilarious and their little graphic is perfect. The risk of harm doesn't go down, but the reward goes up, so the harm just becomes the cost of doing business, justified by the reward. So as the reward gets higher and higher, the amount of harm they're willing to justify goes up. Feels like society in a nutshell.

[soundworlds]

If I understand this correctly, Anthropic's argument is now "yes this will blow up some of your infrastructure, but it will be worth it"

The problem is that no one has been able to prove that it is actually worth the cost. That is a very fragile assumption.

[daveshistory]

It's Shrek logic. "Some of you are going to die, and that is a sacrifice I am willing to make."

[TeMPOraL]

No, it's the actual reasonable approach that sane people have to security. In the real world, security is always about costs and benefits, because you can always make something more secure than it is by spending more money, but it also doesn't make sense to spend more than you're getting from it.

Normally, you secure things up to minimize (${cost of security measures} + ${expected damage from attacks that materialized}), writing off actual material damage with insurance wherever possible. You pick security measures based on their effectiveness, which usually translates to "how expensive will it make success for attackers", aiming to push that above the value the attackers can expect to gain.

There are obvious exceptions to that, like risk to life and limb, as well as some other special situations where attackers may have unusual motivations and thus the economic logic of "make stealing treasure cost more than the treasure" stops applying. But those are exceptions. Almost everything you deal with in your life - from your bike shed to the corporation that owns your bank - follows the above logic in terms of security.

--

I spell this out because I've noticed that tech industry circles have this weird, belief in security as some kind of binary, holy good, that you either have and are blessed, or don't and sin. This obsession starts with failing to even recognize, much less ask, the most important questions about security: why do you want to protect it, and who are you protecting it from?

[hext]

100% agree, and so happy to see somebody call this out. If you go on /r/SelfHosted or any other novice oriented forum, you’ll quickly realize that most users are simply “keeping up with the joneses” when it comes to security & redundancy. That itself is fine I guess, but the zero tolerance they have for anything else is just absurd.

[alansaber]

This has always been the premise. They can't fix the fundamental problems with LLMs but they can continue to optimise them for IE parsing large volumes of data quickly

[jon-wood]

Everything you do a risk/reward equation, you just don't usually see it drawn out quite so starkly. Getting out of bed in the morning carries a risk that you'll trip and crack your head on the floor. Crossing a road carries a risk of being hit by a bus. Eating food carries a risk of choking on it. The same is true in computer security. The only truly secure computer is one you don't turn on, and even that carries some risk of an attacker breaking in and stealing the storage from it.

Whether you agree that the potential harms outweigh the benefits in this case or not those calculations are always happening, so yes, I guess you're right. That is society in a nutshell.

[vrganj]

But if you eat food, I don't risk choking. They want us to take the risk for their reward.

[pixl97]

But if I drive a car, You do run the risk of getting ran over. We can come up with any number of analogies of varying rightness and wrongness here.

[daveshistory]

And then there's a whole truckload of case law about liability that comes into play.

We haven't yet written those laws for "AI."

[hombre_fatal]

What do you have in mind?

You're paying for their services to collect reward for yourself, but also deciding your own risk/reward when choosing e.g. how much access to grant Claude for any given task.

I guess there's the case where the more capable Claude is, the more someone else can use it to find vulns in your services while Anthropic collects their subscription money? But that is mitigable risk that you shipped regardless of what Anthropic is doing.

[6gvONxR4sf7o]

My point wasn’t about risk vs reward, or in their words “harm” vs reward. It’s about how increasing the opportunity for reward increases the justifiable harm. “X is bad (unless it makes me rich).”

I guess it’s the fact that Anthropic usually frame this around morality and risk to society that makes it different. Instead of “risk/harm to me vs reward to me,” their framing reads as “risk/harm to us vs reward to me” or “immorality vs reward to me.” That’s what makes it feel like a great metaphor.

The standard cost benefit analysis we all do justifies increasing the harm to others if the opportunity to benefit ourselves goes up.

[esikich]

Sure. You start a PC repair business. At first, losing a stick of RAM or frying someone's motherboard is super costly when you are doing 10 a week. But once you're doing 1000, that's pretty damn good and easily covered. When you have more tools, velocity, and whatnot, the proportions change.

[altmanaltman]

Wouldn't you lose multiple sticks or fry multiple motherboards as you scale and do 1000? If you're frying 1 at 10, that means you're frying 100 at 1000. Your costs etc will scale as well unless you actually lower the risk/reward ratio, no?

[kuboble]

I think the point is that at small scale a single accident poses a risk of ruin to your small operations.

[chrncirurp]

> I think the point is that at small scale a single accident poses a risk of ruin to your small operations.

At big scale, a single big accident poses a risk to ruin your big operations.

[enraged_camel]

No, it does not. Every large company eventually has a big accident. They survive because they have both the resources (e.g. to fight ensuing legal battles, or pay fines, or simply weather a hit to reputation and the resulting downturn in revenue) as well as redundancy, different types of insurance, and so on.

[zaphar]

They also survive because they invest those resources in some amount of mitigation ahead of time. They don't survive when they don't scale their mitigations along with the business.

[daveshistory]

Companies of all sizes should have insurance to cover such scenarios. You need to get tradesman's insurance on your repair work, or you need to ask yourself why the insurance companies won't insure you.

[truculent]

The point is that if you have a 10% chance of frying motherboard, at 10 a week, you might expect 1 fried p/w, but it could easily be more which may be catastrophic.

At 1000, the number of fried boards will be more predictable and therefore the risk to the business is lower, even if the long-run averages are the same.

[TeMPOraL]

At 1000, you can afford better tools and better employees, and replacement parts get cheaper as you order in bulk, and you can explore clever strategies to smooth risk curves.

At 100 000, you can afford a better and continuously improving process, and dedicated facilities, and skilled experts, and parts get even cheaper because you're a volume buyer or perhaps own the supply side, and you get to set your own risk curve.

Lots of things get cheaper at scale. Insurance, too.

[solenoid0937]

That's how decisions are made IRL. Risk/reward is a thing.

[vrganj]

This is risk to us and reward for them though.

[alansaber]

Exactly. Though with inference cost they're still only making money on enterprise use.

[TeMPOraL]

Because we're all paying for LLM access for shits and giggles, and not because we're getting actual value from it.

[vrganj]

I don't care why you pay for LLM access, it's still spamming my online forums and codebases.

[TeMPOraL]

LLMs don't spam on their own. Take it up with people who wield them.

[ben_w]

They kinda do though, in that instances have been observed to send unrequited messages even when the person/people in charge of some account didn't expressly ask the models to do so.

For my own use of LLMs, I do try to avoid anything which I know has a risk the artefacts they produce may end up DoSing or spamming, and I've avoided the OpenClaw-type pattern for a broader range of reasons of which this is simply one tiny part, but I'm not absolutely confident I could avoid this even in the code coming out of the free tier of the web chat interfaces except by checking every single line of output every single time.

[vrganj]

Nah, it's the technology's fault for enabling it.

[daveshistory]

Many companies would say that's the best kind of risk-reward balance. For them, anyway.

[7e]

They don’t consider risk of ruin and that is where this calculus falls apart. The reward does not reduce the risk of ruin, which increases with blast radius. YOLO!

[heisenbit]

Limited liability makes taking unlimited risks a rational choice. AI ‚only‘ scales this corporate model up and compresses the timeframe to the next disaster.

[keithnz]

but no matter what you do this is the tradeoff you are making. Different people have different tolerances for that balance, hence why I'm happy to watch people on youtube in wingsuits and not do it myself. Of course in this new AI world, quantifying the probability and scale of harm is hard/not fully known. We are trying to mitigate risks with AI, but who knows, could be one misstep away from plummeting off a cliff.

[andai]

Yeah I was thinking about Simon Wilson's "lethal trifecta"[0] in the context of OpenClaw style "general purpose" AI agents, where people just gave it access to their full hard drive, gmail account, etc.

I was thinking you can't make the chance of catastrophic failure zero (we still hear about "Claude deleted my home folder"), but you can definitely limit the blast radius.

You can't get the risk to zero. But the opportunity cost of not playing the game is rising. So you accept some level of risk.

My personal take here is "why screw around with containers and virtualization when a used ThinkPad is $50". Just give it its own machine. Then it can blow it up all it wants. (Or a $3 VPS, as the case may be :)

[0] The lethal trifecta for AI agents: private data, untrusted content, and external communication - https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

[barrkel]

Containment of the execution environment isn't really the issue. It's API tokens that were designed with coarse permission scoping so agents get more power than they need. The risk isn't that your machine gets hacked. It's that your email gets deleted, or forwarded to someone who uses it to break into your other accounts via password recovery.

[zaptheimpaler]

I tried the VPS briefly, it didn't really solve anything for me. The personal assistant agent is only as useful as the data & tools it has, that's where the real risk is. Separate box gives you isolated FS but docker also does that very easily.

[jon-wood]

Docker is not a security boundary. It never has been, but given recent demonstrations of container escapes its even less of one than it ever was. If you want to properly contain a process it needs to be running in a VM of its own, or you need to accept that there's a risk of it escaping and ending up with more access than you planned.

[altmanaltman]

> But the opportunity cost of not playing the game is rising

The opportunity cost of not using OpenClaw? I don't think it's that foundational yet that there is an opportunity cost to not using it. Most people have no purpose for a general-purpose AI both in their personal lives and at work, there is no sense trying out OpenClaw when you don't even know what it'll do.

[charcircuit]

All of ecommerce is built on top of encryption with a non 0 chance of being cracked. The risk is much smaller than the benefit so people are willing to use it and then deal with whatever potential fraud comes from encryption being broken separately.

Technically a merchant could require meeting in person to exchange a OTP to avoid this and make it 0 but it is not worth it and you will get out competed by other businesses willing to take on a marginally higher amount of risk to unlock a lot of utility for the user.

[e12e]

Wiping out a VM, server or workstation should not really be a problem - just restore from backup.

Silently corrupting files, that goes undiscovered until after backup window closes, and data exfiltration are the immediate, serious risks.

[koolba]

> Then it can blow it up all it wants. (Or a $3 VPS, as the case may be :)

Just make sure it doesn’t have ssh access to any other machines!

[chrisweekly]

Is a used Thinkpad really a viable part of your AI workflow? (And is that really a better solution than eg smolmachines microvms?)

[xp84]

I’m a usual booster of AI (others have accused me of being completely in the bag for the clankers) and even I agree fully. These yahoos would clearly give Claude the nuclear launch codes or enough access to copy its full model into the wild if the supposed “reward” promised was large enough.

[daveshistory]

Hardly a new hypothetical scenario, that Wargames movie is probably 40 years old now.

[ronsor]

This is how humans weigh most decisions in practice.

[Frieren]

> the amount of harm they're willing to justify goes up. Feels like society in a nutshell.

Neocon society. Socialism is not like that.

[radu_floricica]

Well, yeah, which is why it's evil. Socialism I mean. How else would you call failing to do basic utility math while insisting you should govern and shape society?

My answer to the trolley problem is that you're allowed to not kill... unless you're the railway manager. If you're in a position of authority you pull the shit out of that switch, and then drink yourself to sleep at night. This is what authority means, not choosing the "feel good, ignore the people that could have been saved" path.

[pjc50]

Running into the problem that Americans are very bad at defining "socialism" here, meaning anything from social democrat to full Communism, but: there is a strong utilitarian streak in socialist societies that is also vulnerable to "the pain (for you) will be worth it (for someone else)" reasoning.

[Frieren]

> there is a strong utilitarian streak in socialist societies that is also vulnerable to "the pain (for you) will be worth it (for someone else)" reasoning.

Socialism is not perfect, it is just better than any other alternative.