[tptacek]

Software security has been running off large-scale automation for over a decade. LLMs might or might not be a step change in that automation (I'm optimistic but uncertain), but, unlike in conventional software development, the standard arguments about craft and thoughtfulness aren't operative here. If there was an argument to be had, it would have been had around the time Google stood up the mega fuzzing farms.

A fun thing to keep in mind about software security is that it's premised on the existence of a determined and amoral adversary. In the long-long ago, Richard Stallman's host at MIT had no password; anybody could log into it. It was a statement about the ethics of locking down computing resources. That's approximately the position any security practitioner would be taking if they attempted to moralize against LLM-assisted offensive computing.

[eyberg]

I kinda see the different side of the coin.

"a determined and amoral adversary" - I'd kinda disagree with this (the amoral adversary part being necessary). If you crawl through the vast data breach notification lists that many states are starting to keep - MA, ME, etc. there are so many of them (like literally daily banks, hospitals, etc. are having to report "data breaches" that never ever make the news) - not all of them are happening cause of ransomware. Sometimes it's just someone accidentally not locking a bucket down or not putting proper authorization on a path that should have it. It gets found/fixed but they still have to notify the state. However, if someone doesn't know what they are looking at, or it's a program so it really has no clue what it's looking at and just sees a bunch of data - there's no malicious intent but that doesn't mean that bad things can't happen because that data has now leaked out.

Guess what a lot of these LLMs are training on?

So while Andrey's software is finding all sorts of interesting stuff there's a bunch of crap being generated inadvertently that is just bad.

[bradyriddle]

Say more about these mega fuzzing farms. I haven't heard anything about this.

[cibyr]

ClusterFuzz: https://google.github.io/clusterfuzz/

[navanchauhan]

Something like https://security.googleblog.com/2011/08/fuzzing-at-scale.htm... ?

[tptacek]

There are fields, endless fields, where kernel zero days are no longer born. They are grown.

[AStonesThrow]

  [e-mail]

It was actually the A.I. Lab at M.I.T. and they already had their own dedicated subdomain for it. This had to have been around 1990-91. And IIRC, the actual admins made a valiant effort to keep all the shell users away from "root" privileges, so it wasn't a total dumpster fire and the system stayed alive, mostly

https://en.wikipedia.org/wiki/MIT_Computer_Science_and_Artif...

[tptacek]

I mean, I remember, in 1994, being on those systems. But it meant nothing. Anybody could be. There wasn't even a glimmer of interestingness about it. It was like "ls"'ing around an anonymous FTP server.

[AStonesThrow]

Hey, I cannot even begin to describe the thrill I got when I first found my way to the AF.MIL anon-ftp server! It was probably sparsely populated with public domain software and a couple boring games, but it felt like I'd just walked in the front gate of Miramar and witnessed the Blue Angels doing barrel rolls.

Sure, it was basically "a poster on the wall" for the US Air Force, and the US Army guy on Usenet shared nothing about his actual Ballistics Research Labs experiments, but for a college freshman kid, I'd never been on a way k00ler bboard, doodz!!1

[anthk]

ITS had no root