[btown]

The problem, though, is that this turns "one of our developers was hit by a supply chain attack that never hit prod, we wiped their computer and rotated keys, and it's not like we're a big target for the attacker to make much use of anything they exfiltrated..." into "now our entire source code has been exfiltrated and, even with rudimentary line-by-line scanning, will be automatically audited for privilege escalation opportunities within hours."

Taken to an extreme, the end result is a dark forest. I don't like what that means for entrepreneurship generally.

[linkregister]

This is a great example of vulnerability chains that can be broken by vulnerability scanning by even cheaper open source models. The outcome of a developer getting pwned doesn't have to lead to total catastrophe. Having trivial privilege escalations closed off means an attacker will need to be noisy and set off commodity alerting. The will of the company to implement fixes for the 100 Github dependabot alerts on their code base is all that blocks these entrepreneurs.

It does mean that the hoped-for 10x productivity increase from engineers using LLMs is eroded by the increased need for extra time for security.

This take is not theoretical. I am working on this effort currently.

[fragmede]

It's great news for developers. Extra spend on a development/test env so dev have no prod access, prod has no ssh access; and SREs get two laptops, with the second one being a Chromebook that only pulls credentials when it's absolutely necessary.

[linkregister]

Yes, having a good development env with synthetic data, and an inaccessible, secure prod env just got justification. I never considered the secondary SRE laptop but I think it might be a good idea.

[wafflemaker]

Please explain the second laptop. I'm studying cybersecurity, so think I should know why. Or is it a joke?

[linkregister]

The value-add is having a workstation that's disconnected from work that would be susceptible to traditional vectors that endpoints are vulnerable to. For example, building software that pulls in potentially malicious dependencies, installing non-essential software, etc. The "SRE laptop" would only have a browser and the official CLI tools from confirmed good cloud and infrastructure vendors, e.g. gcloud, terraform.

I think that such a posture would only be possible in a mature company where concerns are already separated to the point where only a handful of administrators have actual SSO or username/passphrase access to important resources.

[fragmede]

It's not a joke. Supply chain attacks are a thing, but Google Chromebooks are about the most trustable consumer machine you can run custom code on short of a custom app on an iPad. The Chromebook would only ever have access to get the root AWS (or whatever) credentials to delete, say, the load balancer for the entire SaaS company's API/website. If my main laptop gets hacked somehow, the attacker can't get access to the root AWS credentials because the main laptop doesn't have them. The second laptop would only be used sparingly, but it would have access to those root credentials.

[pixl97]

I disagree that it's extra time for security, it's the time we should have been spending in the first place.

[eru]

> Taken to an extreme, the end result is a dark forest.

Sorry, how does that work?

[bryanrasmussen]

since the suggestion is that the new security bug finding LLMs will increase protection because it will have access to the full source code then, the dark forest fear would be, if it is possible for an attacker to get all the source the attacker will be in a better position.

This seems wrong however, as it ignores the arrow of time. The full source code has been scanned and fixed for things that LLMs can find before hitting production, anyone exfiltrating your codebase can only find holes in stuff with their models that is available via production for them to attack and that your models for some reason did not find.

I don't think there is any reason to suppose non-nation state actors will have better models available to them and thus it is not a dark forest, as nation states will probably limit their attacks to specific things, thus most companies if they secure their codebase using LLMs built for it will probably be at a significantly more secure position than nowadays and, I would think, the golden age of criminal hacking is drawing to a close. This assume companies smart enough to do this however.

Furthermore, the worry about nation state attackers still assumes that they will have better models and not sure if that is likely either.

[staplers]

  I would think, the golden age of criminal hacking is drawing to a close. This assume companies smart enough to do this however.

It's rarely the systems that are the weak link, rather the humans with backdoor access.

[btown]

Any single company might be able to proactively defend themselves from attackers, but will companies invest the tokens in this? Most people simply don't care until it's too late.

And in a world where companies begin to suffer from attacks as a result - can the ones who are willing to invest in security defend themselves, not just against cyberattackers, but against a broader investor and customer backlash that believes that startups that build their own technology stacks are riskier due to perceptions about cybersecurity?

An angel investor or LP who sees news articles and media about cyberattacks, then has a portfolio company get hacked in a material way, may simply decide the space has become too risky for further investments, no matter how much prospects get on better security footings.

The dark forest hypothesis, at its core, is about a decision of whether to put your neck out in the universe; if the weapons and countermeasures being used are too horrifying to fathom, the risks unquantifiable, one chooses not to extend one's neck. And that is how an industry begins to dry.

[linkregister]

The pressure by internal auditors and cyber insurance providers to implement these programs will be strong. I have been at organizations where EDR was added only due to the board of directors following the recommendation of 3rd parties. Of course, there will be new companies that haven't achieved the maturity to have had these pressures. But new companies being thoroughly compromised is hardly a recent phenomenon.

[eru]

Does this have anything to do with the other 'dark forest'? https://en.wikipedia.org/wiki/Dark_forest_hypothesis

I don't see the connection.

[bryanrasmussen]

I guess the connection would be human history, a dark forest is a scene of lawlessness and violence and danger in much of that history - at least where stories are concerned.

In the use of the phrase Dark Forest to explain the Fermi paradox it suggests that alien civilizations have kept themselves dark out of fear that the rest of the forest is actually lawless and violent.

In this case though we are entering a dark forest, like Hansel and Gretel, supposedly defenseless against the monsters that lurk in there, but really - they weren't that defenseless were they? I don't think the phrase that apt.

[eru]

> In the use of the phrase Dark Forest to explain the Fermi paradox it suggests that alien civilizations have kept themselves dark out of fear that the rest of the forest is actually lawless and violent.

It's more complicated.

For the Fermi paradox version of the 'Dark Forest' to work, you need civilisations to actively go out and destroy any other form of life they find announcing themselves:

> The "dark forest" hypothesis presumes that any space-faring civilization would view any other intelligent life such as theirs as an inevitable threat and thus destroy any nascent life that makes itself known. As a result, the electromagnetic radiation surveys would not find evidence of intelligent alien life.

Wikipedia has a section on game theory etc.

Without this additional element (basically the version you describe), the dark forest theory doesn't explain the Fermi Paradox: it's just another filter that might perhaps exclude 90% of civilisations, but many civilisations would still be dumb enough to announce themselves. Humans certainly did and keep doing so: it only needs a some people to send a message, and near unanimity to not send anything.

(And that's completely ignoring that our very atmosphere with its chemical imbalance has been sending a strong message of "there's probably life here" for billions of years now. Even our own technology, still in its infancy, is increasingly able to pick up clues about the chemical composition of the atmosphere of exoplanets ever further away from us. And we are still getting better quickly.)

If you add the element that other civilisation are hiding, but come out of hiding just to strike, that breaks down as soon as you have more than two players. Or even just the faint possibility of more than two players.

When you know there are only at most exactly to players, and you are the lurker and find someone else being 'noisy': yes, you have an incentive to strike. When there might be other third parties lurking, you better stay quiet, lest you invite a strike by a third party against you.