[paweladamczuk]

What about the cybersecurity aspect of bespoke software?

A cybersecurity research company can now spend a small fortune on finding zero days in iOS because of the amount of people that use it. It basically guarantees there will be clients like government agencies willing to pay through the nose for the exploits.

Software made for one might disrupt this business model.

[lionkor]

Software made for one, made by LLMs which regurgitate the average of existing tools, are going to have more security issues, not less.

[theshrike79]

But how would you exploit them when every one of them is subtly different?

With software that's deployed to millions of computers you have an abundance of targets, but trying to target some random LLM average todo list at scale is hard, isn't it?

[lionkor]

Yes, but it should be fairly easy to "simply" attack the common technologies that LLMs keep parroting. NextJS, or some Rust tools, or whatever other tools LLMs "love" using, are all great targets.

Once millions of completely unskilled developers have "workflows" that consist of asking an LLM to make a thing, followed by those LLMs pulling in the same 100 (often outdated versions of) dependencies, you have a beautiful attack vector.

Yes, it's "easy" to attack something like Obsidian. It's probably easier to attack a couple hundred dependencies LLMs like to use, or to test what LLMs commonly do to implement things from scratch, and attack those weaknesses.

We are just lucky that enough real, smart, people engineered things that actually work, are well understood, and keep us safe, like firewalls.

[cestith]

Supply chain attacks will still work. Most people aren’t going to have a custom Node, a custom CPython, or all custom libraries. One can fuzz software without the source for common classes of vulnerabilities. With the same handful of models writing a bunch of that bespoke software new horizons open up. Maybe GPT tends to insert the same bug over and over, while Claude inserts another.

Maybe, in fact, some group somewhere combines supply chain attacks with models and/or agents. A model or agent that’s compromised upstream and becomes designed to insert a backdoor is not beyond possibility.

[i_think_so]

I am imagining some poor sod working for NSA TAO trying to hack a bespoke web microservice stack. He spends dozens of hours slaving away at the keyboard, skipping sleep and eating terrible meals at his desk, desperate to get RCE as quickly as possible, because he needs to traverse all the way to the DB layer and exfil data or his boss will pass him over for his next promotion.

At day 9, right as he is getting ready to deploy his beautifully crafted shell code, the clock hits midnight UTC. The website shuts down for maintenance.

"This is it" he thinks. "As soon as the backups finish I'm getting in. No problem."

Minutes tick by. He gets up, stretches, sits back down, watches the clock impatiently. Then, as he prepares to start refreshing the site he recollects, "I'm glad I begged so hard to get authorization to use this PHP 0day."

His partially obscured terminal window has the script ready to launch, all arguments pre-populated, waiting for the link and session token to be pasted in and executed.

The site comes back up. But the url of his launch point returns 404. Undaunted, he returns to a previous url. It is also 404. He curses aloud. Beginning to perspire, he goes to the homepage and prepares to navigate back to the launch point.

The link isn't there. Well, it's there, but it has changed.

"What the....!" The link is no longer a PHP url. He mouses over other links. NO links say PHP anymore. Starting to panic, he clicks on links at random. Not a single one appears to be PHP.

The following morning he schedules an urgent meeting with his supervisor.

"How's that project coming along. Got anything yet?"

"No. I, uh...I'm going to need a bit more time."

"Oh?"

"Yeah. Uh. The site. It got..." He mutes his microphone and, for the 22nd time since midnight, he screams in frustration. Unmuting, he continues:

"It got rewritten. Completely. In Nim."

"What??"

"Yeah. It's some esoteric language that just got a new web framework. I guess somebody decided they wanted to mess around with it. So they vibe coded a complete translation. The whole front end is nimlang now. None of the PHP attacks are going to work on it."

His supervisor expresses his disgust and ends the call.

11 days later the process repeats itself, this time with Rust.

The TAO engineer submits an application to change jobs to the DoD's procurement division, then requests an appointment with a mental health counselor.

[cestith]

Why wouldn’t he use an agent to find the weakness? How would he know what language is on the backend of a web service without already having infiltrated the server? If he’s in the NSA, why wouldn’t he just sneak a vulnerability into common PHP, Nim, or Rust libraries the site is likely to use?

[i_think_so]

Bro. Come on. Don't overthink it.

I wasn't trying to write a Great American [Cyber]Spy Novel. I banged out a silly short story over maybe 40 minutes while I was eating. Then I went back and cleaned up a bit of the text I didn't like and gave it to the world.

I smiled and chuckled a few times as I was writing, because that's what often happens when an author is making something he's happy with. I hoped a few others would get a chuckle out of it too. I gather that 2 people did, so far. That's good enough for me.

If you're applying to become my editor, please email me at bizinquiries@i_think_so.com during M-F so we can discuss your fee.

[cestith]

I’m one of your upvoters. Just making some notes. It’s good fiction, and most fiction has holes when people in the field involved look closely. That’s not a dig at it. I enjoyed reading it enough to engage.

[whytevuhuni]

Moral of the story: A truly secure website would be a continuously morphing one where an LLM keeps rewriting and redeploying large parts of its code every minute, so that no attacker can keep up.

[i_think_so]

Hmm. Now that you mention it, wasn't that part of what was happening in Neuromancer? The "encryption" (or whatever it was) kept changing so the attack had to respond by "evolving" to get in.

Excuse me, I need to go solicit VC for my new evolving web security startup that is really just Claude rewriting 10% of the infra each day....