[OakNinja]

"IBM Bob is IBM’s new coding agent, currently in Closed Beta. "

Promptarmor did a similar attack(1) on Google's Antigravity that is also a beta version. Since then, they added secure mode(2).

These are still beta tools. When the tools are ready, I'd argue that they will probably be safer out of the box compared to a whole lot of users that just blindly copy-paste stuff from the internet, adding random dependencies without proper due diligence, etc. These tools might actually help users acting more secure.

I'm honestly more worried about all the other problems these tools create. Vibe coded problems scale fast. And businesses have still not understood that code is not an asset, it's a liability. Ideally, you solve your business problems with zero lines of code. Code is not expensive to write, it's expensive to maintain.

(1) https://www.promptarmor.com/resources/google-antigravity-exf... (2) https://antigravity.google/docs/secure-mode

[InsideOutSanta]

While they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

[iLoveOncall]

> If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

You don't even need to give it access to Internet to have issues. The training data is untrusted.

It's a guarantee that bad actors are spreading compromised code to infect the training data of future models.

[derektank]

A problem yes, but I think GP is correct in comparing the problem to that of human workers. The solution there has historically been RBAC and risk management. I don’t see any conceptual difference between a human and an automated system on this front

[nkrisc]

> I don’t see any conceptual difference between a human and an automated system on this front

If an employee of a third party contractor did something like that, I think you’d have better chances of recovering damages from them as opposed to from OpenAI for something one of its LLMs does on your behalf.

There are probably other practical differences.

[moron4hire]

A human worker can be coached, fired, terminated, sued, any number of things can be done to a human worker for making such a mistake or willful attack. But AI companies, as we have seen with almost every issue so far, will be given a pass while Sam Altman sycophants cheer and talk about how it'll "get better" in the future, just trust them.

[SoleilAbsolu]

Yeah, if I hung a sign on my door saying "Answers generated by this person may be incorrect" my boss and HR would quickly put me on a PIP, or worse. If a physical product didn't do what it claimed to do, it would be recalled and the maker would get sued. Why does AI get a pass just pooping out plausible but incorrect, and sometimes very dangerous, answers?

[philipallstar]

> Yeah, if I hung a sign on my door saying "Answers generated by this person may be incorrect" my boss and HR would quickly put me on a PIP, or worse

I also have never written a bug, fellow alien.

[premiumLootBox]

I do not fear the employee who makes a mistake, I fear the AI that will make hundreds of mistakes in thousands of companies, endlessly.

[lelandfe]

We need to take a page from baseball and examine Hacks Above Replacement

[conradev]

If anything, the limit of RBAC is ultimately the human attention required to provision, maintain and monitor the systems. Endpoint security monitoring is only as sophisticated as the algorithm that does the monitoring.

I'm actually most worried about the ease of deploying RBAC with more sophisticated monitoring to control humans but for goals that I would not agree with. Imagine every single thing you do on your computer being checked by a model to make sure it is "safe" or "allowed".

[stonogo]

The difference is 'accountability' and it always will be.

[mistrial9]

no, you have a trust problem. Is the tool assisting, or is are the tools the architect, builder, manager, court and bank?

[acessoproibido]

>If you allow a human to edit your code and also give them access to untrusted data (like the Internet), you have a security problem.

Security shouldn't be viewed in absolutes (either you are secure or you aren') but more in degrees. Llms can be used securely just the same as everything else, nothing is ever perfectly secure

[NovemberWhiskey]

Things can only be used securely if they have properties that can be reasoned about and relied upon.

This is why we don't usually have critical processes that depend on "human always does the right thing" (c.f. maker/checker controls).

[OakNinja]

They can be reasoned about and relied upon.

The problem is that people/users/businesses skip the reasoning part and go straight to the rely upon part.

[withinboredom]

They can be reasoned about from a mathematical perspective yes. An LLM will happily shim out your code to make a test pass. Most people would consider that “unreasonable”.

[strken]

I have an issue with the "code is a liability" framing. Complexity and lack of maintainability are the ultimate liabilities behind it. Code is often the least worst alternative for solving a given problem compare to unstructured data in spreadsheets, no-code tools without a version history, webs of Zapier hooks, opaque business processes that are different for every office, or whatever other alternatives exist.

It's a good message for software engineers, who have the context to understand when to take on that liability anyway, but it can lead other job functions into being too trigger-happy on solutions that cause all the same problems with none of the mitigating factors of code.

[Eufrat]

> When the tools are ready, I'd argue that they will probably be safer out of the box compared to a whole lot of users that just blindly copy-paste stuff from the internet, adding random dependencies without proper due diligence, etc. These tools might actually help users acting more secure.

This speculative statement is holding way too much of the argument that they are just “beta tools”.

[cyanydeez]

You would think so, but you should read about how they bear proof trash cans in yellow stone.

They cant. Why? Because the smartest bear ia smarter than the dumbest human.

So, these AIs are suppose to interface with humans and use nondeterminant language.

That vector will always be exploitable, unless youre talking about AI that no han controls.

[OakNinja]

Yes. But the exploitable vector in this case is still humans. AI is just a tool.

The non-deterministic nature of an LLM can also be used to catch a lot of attacks. I often use LLM’s to look through code, libraries etc for security issues, vulnerabilities and other issues as a second pair of eyes.

With that said, I agree with you. Anything can be exploited and LLM’s are no exception.

[cyanydeez]

As long as a human has control over a system AI can drive, it will be as exploitable as the human.

Sure this is the same as positing P/=NP but the confidence that a language model will somehow become a secure determinative system fundamentally lacks language comprehension skills.