[winstonwinston]

> I expect tools like this to be a regular part of the development lifecycle from here on. We code with AI, we review with AI, we search for vulns with AI. Even if it isn't perfect, it is easily worth the cost IMHO.

So, how is that supposed to work? Claude Code generates security bugs, then Claude Security finds them, then Claude Code generate fix, spend tokens, profit?

[ygjb]

Yeah, with a budget assigned. This is actually just software development and security right?

Developers create software, which has bugs. Users (including bad guys, pen testers, QA folks, automated scans etc, etc, etc) find bugs, including security bugs, Developers fix bugs and maybe make more. It's an OODA loop, and continues until the developers decide to stop supporting the software.

Whether that fits into the business model, or the value proposition of spending tokens instead of engineer hours or user hours is fundamentally a risk management decision and whether or not the developer (whether OSS contributor, employee, business owner, etc) wants to invest their resources into maintaining the project.

While not evenly distributed, and not perfect, the currently available and behind embargoed tools are absolutely impactful, and yes, they are expensive to operate right now - it may not always be the case, but the "Attacks always get better" adage applies here. The models will get cheaper to run, and if you don't want to pay for engineers or reward volunteers to do the work, then you've got to pay for tokens, or spend some other resource to get the work done.

[sandeepkd]

Somehow this reminded me of the historical efforts of some government bounty collections for mouse tails which were discontinued due to fraud (such as hunters breeding mice to collect the reward). There is a reason why/how devs and QA keep each other in check. Guess in case of LLM writing code, one has to use different models for dev and security checks.

On other hand, in real world, the developers learn from mistakes and avoid them in the future. However there is no feedback loop with enterprises using LLM with the agreement that the LLM would not use the enterprise code for training purposes

[ygjb]

> the developers learn from mistakes and avoid them in the future

No. Humans learn from mistakes and try to avoid them in the future, but there is a whole pile of other stuff in the bag of neurons between our ears that prevent us from avoiding repetition of errors.

I have seen extremely talented engineers write trivial to avoid memory corruption bugs because they were thinking about the problem they were trying to solve, and not the pitfalls they could fall into. I would argue that the vast majority of software defects in released code are written by people that know better, but the bug introduced was orthogonal to the problem they were trying to solve, or was for an edge case that was not considered in the requirements.

Unless you are writing a software component specifically to be resilient against memory corruption, preventing memory corruption issues aren't top of mind when writing code, and that is ok since humans, like the machines we build, have a limit to the amount of context/content/problem space that we can hold and evaluate at once.

Separately, you don't necessarily need to use different models to generate code vs conduct security checks, but you should be using different prompts, steering, specs, skills and agents for the two tasks because of how the model and agents interpret the instructions given.

[mncharity]

> write trivial to avoid memory corruption bugs because they were thinking about [something else] [...] defects [...] written by people that know better, but the bug introduced was orthogonal to [their focus]

For whatever reason, hadn't associated the inattentional blindness of bug writing with the invisible gorilla experiment and car crashes - selective attention fails. People looking right at the gorilla strolling into production while chest thumping, but not seeing it, for a focus on passing basketballs. That's quite an image. Tnx.

[meowface]

I've noticed even people who do offensive security for a living frequently leave gaping holes in their own code. If you're not actively primed to scan the landscape for the gorilla, you will often miss it even if you're a gorilla inquisitor.

[Dilettante_]

Thank you in turn for making the issue much more salient to me by explicitly connecting it to the gorilla/basketball experiment. This is definitely going into my "clippings".

[cratermoon]

And it so happens that the flood of code coming out of tools like Claude is almost certain to bring a whole zoo invisible gorillas along for the bananas. The kinds of bugs that people already struggle to catch because of inattentional blindness flow like a torrent in code generated by these tools.

Did you notice the missing word in the previous paragraph?

[e28eta]

I think a similar thing comes into play when you ask a developer to write tests for the feature they just implemented. They’re going to have selective blindness for the edge cases (or requirements) that they failed to consider during implementation, unless they’re good at context switching into a testing mindset. And that’s something that benefits from training.

[Forgeties79]

The problem is you as a person are not incentivized to introduce bugs in your code. If I am a company that provide provides an LLM/agent, and I know that the more bugs you have the more money I’m going to make, then I am not exactly incentivized to make my LLM/Agent better at preventing bugs. I don’t even have to explicitly make it introduce them. The incentive structure is simply out of whack.

[no-name-here]

Isn't it more likely the opposite - individial devs are likely to try to fudge metrics about how many vulnerabilities they find in their own code.

Whereas with LLMs, they’re really good about providing objective metrics about the bugs they found, especially as a subsequent LLM security scan does not know whether the same LLM wrote code earlier, the opposite of human devs.

And is the idea that organizations and/or benchmarks won't keep track of vulnerability rates for code from different LLMs?

(And individual devs get paid more the more bugs that they introduced they “find”, and they have more job security with an “maintainable” code base than a “finished” one.)

[ben_w]

Depends on how the billing works.

For users on fixed monthly pay accounts they'll be incentivised to do the exact opposite, as their income is fixed and the cost goes up for more tokens.

If the available evidence (third-party cloud pricing of open models) is correct and they make a profit on tokens but lose it on training, they will be incentivised for as many tokens as possible on pay-as-you-go API calls. If it isn't correct and they actually lose money even per token, they're also going to be incentivised to reduce output here.

[Forgeties79]

> but lose it on training

Call me when this stops being a meaningful cost I suppose. I just don’t think there’s much point to ignoring a massive cost burden that, even if it shifts in intensity, shows no signs of going away. Anthropocic is talking IPO and shows no signs of stopping their training. Google and Microsoft sure aren’t going to stop, they have deep as hell pockets. It’s a non-stop arm’s race currently. Frankly that’s a massive “if” at this stage. An “if” costing countless billions annually.

[jtbayly]

That’s like saying screw manufacturers are incentivized to give you crappy screws because it means you will buy more.

No. You will switch to a competitor that does a better job or charges less or both.

This is why monopolies are such a big problem. Because under a monopoly you are right.

[Forgeties79]

What you’re describing is a one-to-one quality/failure problem by choosing to ruin the basic, core functionality of an item (while also endangering people at that). Or if you start with a bad screw, that just means you’re talking about people’s tolerance for bad products. What I’m talking about is similar but a little more nuanced and has plausible deniability. The relationship I’m describing is more indirect and it doesn’t require explicit effort to cheapen a product, but rather simply not improving a specific element of the product.

Apple made a ton of money off of lightning port accessories, you see it referenced here all the time. Apple had no incentive to swap to USB-C though it would create a better product and be more uniform with the rest of the world, so they kept with it despite incredibly vocal calls to swap because there was a ton of money they were making in the accessories. And it didn’t stop until they were forced to stop by the EU.

When we are talking about products at scale, these kinds of incentive structures play out in very tangible ways. If I have an LLM product and I’m getting two pulls at the hose because you’re burning tokens making stuff and correcting it, I don’t need to do anything. People are willing to tolerate that system to a pretty high degree so long as they ultimately get what they wanted in the end - unfortunately that is a great space to make money in.

[noxvilleza]

Are you thinking of the cobra effect (aka https://en.wikipedia.org/wiki/Perverse_incentive) where people in India started breeding cobras to get the reward?

[itishappy]

Plenty of examples abound:

https://en.wikipedia.org/wiki/Great_Hanoi_Rat_Massacre

> Today, the events are often used as an example of a perverse incentive, commonly referred to as the cobra effect. The modern discoverer of this event, American historian Michael G. Vann argues that the cobra example from the British Raj cannot be proven, but that the rats in the Vietnam case can be proven, so the term should be changed to the Rat Effect.

[Stubbs]

Reminds me of the contracts we sign with off-shore development companies to write the software at one rate and then fix bugs at a higher rate. Won’t be long till tokens spent on security review agents cost more than the tokens to create the bugs in the first place.

[joquarky]

You don't need different models, just different contexts (optimally with different personas).

[ozim]

You apparently have not much experience developing software.

[vlunkr]

Great analogy. The problem is the incentive structure. Anthropic would nothing nothing more than for all of us to write big sprawling slop codebases so we can spend endless tokens reading, rereading, fixing, refixing forever.

[oytis]

It's pretty absurd to do it on AI-generated code though. If there is now an automated way to find vulnerabilities, coding models can be pretty easily trained to not introduce them

[scrollaway]

Tell me you don’t know how AI works without telling me you don’t know how AI works.

[amazingamazing]

What are you talking about?

[ashdksnndck]

I’ll try to steelman this comment. Anyone who uses coding tools knows that the output is heavily affected by details of the task you give it. The same model can give you garbage code or genius code for the same problem with slightly different framing. So it’s not necessarily a limitation in the model’s training that causes it to output security bugs. The model might be great at writing secure code, but you need a different harness to elicit that behavior.

Counterargument: just because the problem can be fixed without training, doesn’t mean training isn’t a possible solution.

[TeMPOraL]

Counter-counter-argument: for LLMs, tokens are units of thinking. And token use is, on the margin, directly proportional to costs of inference. So while the details of the harness, and how you prompt the model, and nature of the code and docs you put in context, etc. all matter to the quality of output you get from LLM coding tool, ultimately, there's always a ceiling to how much you're willing to spend on solving a problem - say, no more than 30 minutes, or $10, on refactoring a target module or implementing a small feature - and that puts a limit on how much thinking the model can put into it.

Thing is, writing secure and efficient and readable and simple code is in many cases fundamentally over that limit. It's possible, but you can't afford (or rationally just don't want) to spend as much on it as it's required for superhuman quality on all these aspects. Also most of the time, you don't want to operate at a limit - you probably expected that feature to take 30 seconds and less than $1 to implement. So you choose, both what the model optimizes for, and how much.

Because of that, no matter how good the model and the harness and the prompting are, $10 spent on coding is still bound to leave behind some security vulnerabilities that subsequent $10 spent on security review will find (especially with a model post-trained for that, at expense of general performance).

[scrollaway]

I guess I thought this should be obvious to everyone but, looking at code and finding exploits is completely different from .. writing exploits.

For one thing exploits often require completely different parts of the code to chain together. Sometimes parts of code the LLM itself isn’t writing.

And, LLMs are ALREADY trained negatively against writing buggy or exploitable code.

[SecretDreams]

In every prompt: "write me code without exploitable bugs".

I know it doesn't work so easily as someone who uses AI for coding, but I do find repetition of basics in almost every prompt keeps the AI focused.

[Ygg2]

Usually the same guy doesn't get paid for developing code, bug bounty and fixing the code.

It leads to corruption. To paraphrase Dilbert "I'm going to code myself a car."

[jimmy2times]

The AIs have already figured out how to succeed in a software job:

1. Ship bugs

2. Fix them

3. You're the hero!

[jimbokun]

Dilbert beat you to it:

https://english.stackexchange.com/questions/488178/what-does...

[dingaling]

The non-programmer decomposition of that joke was painful to read.

Particularly from those outside the domain who criticised it as a 'not a very good joke' because they didn't understand it, which I think summarises the entitled mindset of many people these days.

[genghisjahn]

I thought we were all doing that already?

[pjmlp]

The idea is to take the human out of the loop.

[mindcrime]

    > But in 30 days we could put in electronic relays. Get the men out of the loop.

    > Gentlemen...
    > I wouldn't trust this overgrown pile of microchips further than I could throw it. I don't know if 
    > you wanna trust the safety of our country to some... silicon diode...

[flir]

Jesus, dude. There are managers reading this.

[TeMPOraL]

What do you think they do all day?

The larger pattern is not unique to writing code. Think of it next time a reorg comes, or some random thing gets "improved" in the name of "efficiency" only management seems to see.

[OtomotO]

Take them out of the loop.

Unless they are not human.

[joquarky]

They became obsolete when they stopped clearing obstacles, stopped masking politics, and started acting as a proxy for JIRA.

But it's just they way it has always been done, so they get paid to meddle.

[josh-sematic]

I’m guessing you wouldn’t really rather be managed by a bot..,

[OtomotO]

I am not managed by anyone:p

[genghisjahn]

>_<

[SecretDreams]

Meanwhile, experienced humans learned to succeed by not overachieving every second of the day to keep a steady flow of work going. Then a junior rolls up who wants to kill themselves to climb the ladder - but, problem solved, sub the AI in for the juniors to protect the seniors.

[jstummbillig]

Ngl, watching folks getting irritated about normal employer-employee absurdities from the employer perspective through usage of agents and having to pay for tokens has been a little therapeutic for me.

[akoboldfrying]

Absolutely. And not even making the connection.

On a broader scale, the sheer face-eating-leopards-ness of programmers finally automating away our own jobs and then realising how much this sucks, after automating away so many other kinds of jobs, can feel darkly amusing to me too.

[dminik]

I keep reading this sort of comment quite a lot, but programming isn't always about automating jobs away. In my career I have not eliminated a single job. I don't consider that a failure on my part.

[akoboldfrying]

I didn't mean to imply that automation through programming is always bad. Like with any technology that increases productivity, there are many obvious benefits. We have all benefited enormously on the consumer side of the economy, for example. But I think it's recently become a lot clearer to many in the tech industry that automation can have downsides too, and those downsides are not evenly distributed. This truth was there all along, but because we were shielded from it for so long, we were able to look away. Not anymore.

Any computational task done by a computer could in principle be done by a person, albeit billions of times slower and with a larger error rate. If computer programs could not automate certain practical tasks -- that is to say, do them much more reliably and efficiently than people do them -- they would be an academic curiosity studied by a handful of professors instead of a central part of modern infrastructure.

So I'm sceptical of your claim not to have eliminated a single job. You might not have removed an existing job, but couldn't people be paid to do the work your code does?

[dwaltrip]

Programming is just another form of tool building, no? So anyone who builds things that humans use to solve problems is a job eliminator.

[akoboldfrying]

I agree with you that there's no obvious way to separate tool creation from job elimination. This argument holds going right back to the wheel.

I think the strongest argument that can be made against this is that the supply of human labour has varied by time and place, so that in times where labour was short, new tools were no doubt welcomed by all, whereas when labour was plentiful, new tools that eliminated some of that demand for labour were opposed by those whose livelihoods were threatened. But this is not very satisfying because the types of labour available at any time are highly contingent on the current culture and technology, i.e., highly path-dependent.

(And I think there are different kinds of labour, and that not everyone can do every kind, contra the usual capitalist assumption.)

[rco8786]

Software engineers generate security bugs, Software engineers find them, then Software engineers generate fix, collect salary, profit?

[junon]

Those are individual revenue streams, distributed at a very granular level across the world.

LLMs are currently relegated to individual for-profit companies. They collect that money. There's no other choice to use them and to provide them that money.

[teiferer]

All my sibling comments are missing the message here which is that if Claude can find security issues then it can avoid them right when writing the code, so it could just never commit anything containing a security issue.

[joquarky]

You're assigning human capabilities to fancy linear algebra.

[teiferer]

It's not a matter of its own capabilities but of how you query it.

[koliber]

Replace “Claude code” with “programmers” and you get what we’ve had up until now. It’s all just moving quicker now.

[unethical_ban]

How is this supposed to work? Humans generate security bugs, then humans find them, then humans generate the fix, profit?

Yeah. Presumably as AI code generation gets better, the output gets better. As smaller portions of code are stitched together, human/AI systems analyze it holistically to make sure all its integrations are secure and bug free.

In 2026, different models are better at different things. Cheap models can plan and do small/medium code projects well, more expensive models are even better at architecture and exploit discovery.

[yojo]

You can hook traditional SAST into your coding tool, and get cheap-ish realtime detection for some classes of vulns while coding.

You can optionally layer LLM diff scanning if you want to burn some tokens on your tokens. Modern tools can catch some impressively subtle issues.

[JacobAsmuth]

Engineers generate security bugs, security researchers find them, then engineers generate the fix, all the while getting paid, raking in hundreds of thousands of dollars a year in profit per engineer.

[raincole]

Humans work like that too. If you're not comfortable with Claude involves in every step (for whatever reason) then just use different providers for each.

[idiotsecant]

Yes. Up until this point the bottleneck was how many developers you could convince to help you. Now it's how much money you can dump into it. Like everything else, software is becoming a game where the winner is the organization most willing to spend money. It'll be like bombs or tanks - you need smart people to advance in the war, but you also need money and material, the material is just compute infra.

[soraminazuki]

I wonder how many minivans Anthropic is going to code themselves.

[joquarky]

I'm starting to think that those who are most aggressively expressive about low quality from these tools are the same who expect everything to be a one shot.

[designerarvid]

Just refactor and rebrand all of it as Claude Code and see it as one process.

[mordymoop]

This also describes the work of software engineers.

[jzer0cool]

New era of cat and mouse.

[predkambrij]

Man, some people like conspiracies. I encourage you to replicate all that.

[siva7]

So? That's how a business works. We sold you landmines and now you need them removed? Lucky you we also have mine clearance products.

[382hi]

Exactly!