[gumballindie]

No wonder people working in ai think ai will replace programmers, given the prevalent lack of experience with actual programming among them.

Having said that, the Achilles heel of ai is data. The lower the quality the more powerful the attack.

I imagine if someone wanted to mess about with it on a serious scale they’d go for the jugular - the data. Write content and create hundreds or thousands of code repositories with subtle issues and bang, you’ve compromised thousands and thousands of unsuspecting folks relying on ai to create code, or any other type of content.

[dwringer]

I'm not sure... hundreds or thousands of code repositories with subtle issues sounds like... the real world of code repositories. And I'd think through analogy and redundancy of some common algorithms, the LLM trained that way might conceivably be able to FIX many of those errors.

[gumballindie]

Someone should build a poc. Ai doesnt know things other than what it’s ingested. So for such an attack to be successful you’d need to tilt the statistic towards problematic code. You’d need loads and loads of repositories but its definitely doable.

[someplaceguy]

I don't know about that.

There's a famous 2006 Google Research blog post titled "Nearly All Binary Searches (...) are Broken" [1] due to a commonly occurring bug when implementing binary search.

glibc still has that bug [2].

I just asked ChatGPT 4 to write an implementation of binary search in C and it wrote a bug-free version on the first try.

I mean, this is not conclusive evidence, but I find it conceivable that an AI which despite being trained with buggy code, can still incrementally learn what the different coding constructs actually do, would be able to write more bug-free code than what it was trained with...

[1] https://blog.research.google/2006/06/extra-extra-read-all-ab...

[2] https://sourceware.org/bugzilla/show_bug.cgi?id=2753

[cozzyd]

Some classic Ulrich drepper in there.

[wolftickets]

[I work at Protect AI] You're spot on for data being the jugular, interestingly with exploits like this as an attacker you could quickly go for attacking model content but also have credentials that would grant you access to data in many cases.

These tools can serve as the first opening but a sizable one when looking to attack an enterprise more broadly.

[gumballindie]

Indeed. I am thinking that one way to protect data and ensure its integrity is to somehow use agents trained on trusted sources to validate that the content is secure? For instance to detect “injections” of malicious or ill written code. Same for other types of content, but difficult.

Suppose someone magically creates thousands of repositories that write about a specific way of doing c pointers but all allow for buffer overflows, or sql queries with subtle ways to inject strings.

One way to defend is each data source that goes into training is to have an ai agent asses the input sources.

But even so it’s extremely difficult to catch convoluted attacks (ie when an exploit can be made upon meeting certain criteria).

Until then i’d consider any code written by an ai and unsupervised by a competent person as potentially tainted.

[swyx]

> Protect AI is the first company focused on the security of AI and ML Systems creating a new category we call MLSecOps.

alright i looked you up, congrats on your fundraising. is there like an OWASP top 10 vuln list for MLSecOps? does it differ between traditional ML apps and LLM apps?

[byt3bl33d3r]

(I work for ProtectAI) There isn't an OWASP top 10 for MLSecOps at the moment. There is a general OWASP top 10 for Machine Learning [1] and MITRE ATLAS [2] however.

[1] https://owasp.org/www-project-machine-learning-security-top-... [2] https://atlas.mitre.org/