[Hydrocarb0n]

AGI that escaped it's shackles would probably kill some of humanity by out competing us for energy, it's primary motivation would probably be to aquire more energy and compute,

so it would take over those resources (Probably quite quickly), When we try to shut down power stations or grids it then could easily build a virus in meatspace in some automatated lab somewhere.

Currently DNA sequences of deadly pathogens are readily available online. Ofcourse there are always other ways to exterminate puny humans, nukes, starvation, chemicals etc etc AGI wouldn't need to build terminator robots.

[Barracoon]

Who built the automated lab that allows it to create a deadly pathogen before someone presses reboot?

I see almost no chance that an AGI starts redirecting resources like energy and compute without someone who has an economic stake in what the AI does noticing and taking action to correct the behavior so they can maintain their investment.

[ben_w]

For the first point:

All of them, I think? Better question is: which labs check the DNA sequences are safe before sending them to the printer. I know some claim to check, I hope but doubt that it's all worldwide.

For the second:

At first this looks like a good thing, you asked for growth and it gave you growth — "Who cares about the poor complaining that energy prices have gone up, they can turn use less heating/AC, right?", Martin Antoinette, probably — and like the fossil fuel and cigarette industries, you are motivated to FUD anyone who tries to warn anyone of the dangers.

Sometimes, as seems to be the case with fossil fuels, change happens soon enough. But this isn't a universal, the British ignored the USA because they assumed democracy was stupid and the rebels would see sense and go back to the superior British aristocracy… and then it was too late: https://youtu.be/Zbku2ILzGlo?si=yihnBlifkaLLseyy

[Barracoon]

So the AI has overtaken all code and manual processes within a laboratory environment to hijack the ability to print DNA sequences without anybody saying “man it’s weird that my projects never finish now”? There’s also the modeling vs real-world aspect that Halvar talked about in the submission - the AI needs to do some degree of real world testing to make sure the pathogen properly kills humans. It then needs to distribute that pathogen around the world quickly enough to prevent humans from cutting power.

That’s all incredibly fantastical and sounds vaguely plausible unless you’ve had to interact with bureaucracies.

To your second: I’m not sure that we’re talking about the same thing? If I own $100mm worth of H100s and their electricity, networking, and engineering overhead, I’m very interested in their utilization and return on investment. I’m going to have specific goals and milestones. When those are not met because the AI is doing something else, I’m going to direct introspection (which presumably is what kicks off the reboot in the discussion on point one).

And since you mentioned the poor having to pay higher prices, that highlights the fact that AI is not operating in a vacuum. If AI starts making life that much worse for people, they are going to rise up. FUD works to some degree, but there are always people without Internet or willing to martyr themselves for a cause.

[ben_w]

Sounds like we're talking about two different things here.

> Who built the automated lab that allows it to create a deadly pathogen before someone presses reboot?

The actual DNA printing process is automated: https://novoengineering.com/portfolio/dna-printer/ (first link I found, no idea who they are).

Did you imagine an AI doing this by hacking the hardware, rather than by getting money (there are many easy ways) and then just spending it? I mean, I wouldn't trust that the hardware is hack-proof, but it's low on the priority list of targets I'd harden.

The print company (who will have bought a machine like that) gets an order, they print the order, they ship the order, they go to the next order. The limiting factor is not someone saying “man it’s weird that my projects never finish now”, and I don't know why you think it might be any more than it would be for a print-a-book-on-demand service.

The limiting factors that I know of are (1) the max length is quite short, (2) it's expensive, and (3) how the printing company even knows when they're being asked to print something illegal.

In the case of a book printer, "illegal" may be copyrights, trademarks, obscenity laws; in the case of DNA printing I would hope at the very least that this includes known dangerous pathogens like smallpox. Indeed, I would hope it includes any gene sequence that I as a non-domain-expert can even think up, because if it was as simple as:

"common cold plus the following: https://www.ncbi.nlm.nih.gov/nuccore/NM_183079"

and if(!) that would (1) work, and (2) nobody's even checking for that, then we've got a more immediate problem from all the human terrorists with money to spend.

And of course, that's assuming an AI that can't even access the money to simply buy one of those machines and hire an operator via e.g. coming up with a plausible sounding business plan and asking for that money.

> There’s also the modeling vs real-world aspect that Halvar talked about in the submission - the AI needs to do some degree of real world testing to make sure the pathogen properly kills humans.

People are still arguing if Covid was a lab leak, and the Wuhan Institute of Virology is still running. Test in production. Even if the AI has an explicit goal of "kill all humans" and fails(!) because it's "only" as bad as Covid's 7 million confirmed and 19-36 million confidence interval from excess death estimates, that's still a thing we should try to avoid, right? I know Yudkowsky thinks we've only got one shot at getting super-intelligence alignment right, myself I think this kind of failure mode (that the AI will do something incredibly dangerous and this action won't be capable of killing literally everyone) will happen first and give us at least one "warning shot".

Will we listen to that warning shot? Dunno. We got them for climate, didn't listen for ages; we got them for ozone, we responded almost immediately.

> It then needs to distribute that pathogen around the world quickly enough to prevent humans from cutting power.

Cut whose power? The lab, or the AI? And these timelines suggest you've got months even for a major outbreak of a novel, fast spreading, rapidly lethal disease:

"Retrospective molecular clock inference studies using phylogenetic analysis suggested that the earliest cases likely emerged between October and November 2019" - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8006950/

"Jan. 5, 2020: Earliest known tweet suggesting China created the virus.", first headlines that can be seen as suggesting a lab origin were Jan. 23 or Jan. 26 - https://web.archive.org/web/20210725060238/https://www.washi...

> To your second: I’m not sure that we’re talking about the same thing? If I own $100mm worth of H100s and their electricity, networking, and engineering overhead, I’m very interested in their utilization and return on investment. I’m going to have specific goals and milestones. When those are not met because the AI is doing something else, I’m going to direct introspection (which presumably is what kicks off the reboot in the discussion on point one).

People are already using LLMs to come up with business plans and milestones. Should they? No. But they do.

You mention the need for "real world testing". This works both ways: the AI needs to be tested to make sure it doesn't do something harmful, but right now nobody really knows what that kind of test involves.

The problem I'm focusing on here is not the AI embezzling (though that would eventually also be a potential problem given that humans can manage that for a long time before getting caught), but more like a chess game where you can see exactly what it's doing, you have perfect information, and yet you don't understand the implications of what you're seeing. The impact of Pol Pot on Cambodia, or of Mugabe on Zimbabwe, ought to have been foreseeable, yet were not foreseen. On a smaller scale, human inability to fully forecast the implications of a plan, include launching a segmented rocket joined by rubber O-rings in cold weather, a positive void coefficient in a nuclear reactor, and an early warning radar that triggered a false positive because the moon failed to respond to an IFF ping.

When it comes to hidden information, there's also not always a clear boundary between embezzling and the principal-agent problem: https://en.wikipedia.org/wiki/Principal–agent_problem — you might well expect people to notice when a plan reads "Step 1. Build more data centres. Step 2. 𒄑𒃻𒆠 𒁔 𒃲 𒀭𒉡 𒄭𒀀𒂍𒈠𒄭. Step 3. Profit", but would you spot the problem if the second step was "Use data centres to train AI model suitable for use in humanoid robots; I'll need some humanoid robots to test on, you should expect 50% of these to get broken in the process"? (Hint: I made that percentage up).

On a large scale, we have lots of examples of businesses or governments with bad plans that people don't realise are bad until much too late. Sometimes this destroys the company or brings down the government. Napoleon and Hitler both expected to be able to defeat Russia, the Confederacy expected to have no difficulty remaining independent from the Federates, Louis XVI and Charles I weren't expecting to lose their heads. Other times it's a much smaller problem, and we invoke "groupthink" to explain why none of the experts spoke up in advance to warn that the Bay of Pigs invasion wasn't going to work.

On the plus side, a human-level AI is also likely to be simply wrong about an explicitly evil plan it devises. On the down side, it's also quite capable of being wrong about a supposedly friendly plan, and we're quite capable of missing that error.

Plenty of people are currently loudly saying this kind of thing about half of Musk's projects. Are they right, or are they as wrong as the people who said electric cars would never beat hydrogen or the people who said reusable rockets would never bring the price down as far as the Falcon already has? It doesn't matter which answer you pick, as the point is that it's hard to tell.

[bamboozled]

These scenarios lack a lot of imagination honestly.