[ofjcihen]

Worked a contract where this succeeded in pushing through a fail open design.

It also should be a warning to everyone that these groups are now aware of analysis and deobfuscation using AI and to take using a sandboxed environment more seriously.

I’ve personally had about 20% success rate getting opus 4.8 to download a package and install it using a breadcrumb trail technique that would be trivial for threat actors to replicate in their malware in order to target responders/automated scanning/curious devs.

[dcrazy]

What do you mean by “this succeeded?” Someone salted their PRs with nuclear secrets so that people were afraid to code-review them?

[ofjcihen]

No. The intention is most likely to get automated LLM based code review mechanisms to stall out.

Normally you’d want that to result in a fail and a subsequent rejection.

But because the team who made the review agent and pipeline in my example had many false positives at first they resorted to a fail-open and report setup (not uncommon).

So when the LLM hit this bit and then stalled out the pipeline pushed the code to their Artifactory repo anyway resulting in it being used internally -> exfil of secrets and repos etc.

It’s more about bad design but bad design is pretty common unfortunately.

[rcbdev]

This sounds absolutely horrible, in all aspects. Sounds like there is no engineering culture at all.