[ben_w]

Oh, but it is an AI risk.

The analogy is with stock market flash-crashes, but those can be undone if everyone agrees "it was just a bug".

Software operates faster than human reaction times, so there's always pressure to fully automate aspects of military equipment, e.g. https://en.wikipedia.org/wiki/Phalanx_CIWS

Unfortunately, a flash-war from a bad algorithm, from a hallucination, from failing to specify that the moon isn't expected to respond to IFF pings even when it comes up over the horizon from exactly the direction you've been worried about finding a Soviet bomber wing… those are harder to undo.

[A_D_E_P_T]

"AI Safety" in that particular context is easy: Keep humans in the loop and don't give AIs access to sensitive systems. With certain small antipersonnel drones excepted, this is already the policy of all serious militaries.

Besides, that's simply not what the LW crowd is talking about. They're talking about, e.g., hypercompetent AIs developing novel undetectable biological weapons that kill all humans on purpose. (This is the "AI 2027" scenario.)

Yet, as far as I'm aware, there's not a single important discovery or invention made by AI. No new drugs, no new solar panel materials, no new polymers, etc. And not for want of trying!

They know what humans know. They're no more competent than any human; they're as competent as low-level expert humans, just with superhuman speed and memory. It's not clear that they'll ever be able to move beyond what humans know and develop hypercompetence.

[ben_w]

> With certain small antipersonnel drones excepted

And mines, and the CIWS I linked to and several like it (I think SeaRAM is similar autonomy to engage), and the Samsung SGR-A1 whose autonomy led to people arguing that we really ought to keep humans in the loop: https://en.wikipedia.org/wiki/Lethal_autonomous_weapon

The problem is, the more your adversaries automate, the more you need to automate to keep up. Right now we can even have the argument about the SGR-A1 because it's likely to target humans who operate at human speeds and therefore a human in the loop isn't a major risk to operational success. Counter Rocket Artillery Mortar systems already need to be autonomous because human eyes can't realistically track a mortar in mid-flight.

There were a few times in the cold war where it was luck that the lack of technology that forced us to rely on humans in the loop, humans who said "no".

People are protesting against fully autonomous weapons because they're obviously useful enough to be militarily interesting, not just because they're obviously threatening.

> Besides, that's simply not what the LW crowd is talking about.

LW talks about every possible risk. I got the flash-war idea from them.

> Yet, as far as I'm aware, there's not a single important discovery or invention made by AI. No new drugs, no new solar panel materials, no new polymers, etc. And not for want of trying!

For about a decade after Word Lens showed the world that it was possible to run real time augmented reality translations on a smartphone, I've been surprising people — even fellow expat software developers — that this exists and is possible.

Today, I guess I have to surprise you with the 2024 Nobel Prize in Chemistry. Given my experience with Word Lens, I fully expect to keep on surprising people with this for another decade.

Drugs/biosci:

• DSP-1181: https://www.bbc.com/news/technology-51315462

• Halicin: https://en.wikipedia.org/wiki/Halicin

• Abaucin: https://en.wikipedia.org/wiki/Abaucin

• The aforementioned 2024 Nobel Prize for AlphaFold: https://en.wikipedia.org/wiki/List_of_Nobel_laureates_in_Che...

PV:

• Materials: https://www.chemistryworld.com/news/ai-aids-discovery-of-sol...

• Other stuff: https://www.weforum.org/stories/2024/08/how-ai-can-help-revo...

Polymers:

• https://arxiv.org/abs/2312.06470

• https://arxiv.org/abs/2312.03690

• https://arxiv.org/abs/2409.15354

> They know what humans know. They're no more competent than any human; they're as competent as low-level expert humans, just with superhuman speed and memory. It's not clear that they'll ever be able to move beyond what humans know and develop hypercompetence.

One of the things humans know is "how to use lab equipment to get science done": https://www.nature.com/articles/s44286-023-00002-4

"Just with superhuman speed and memory" is a lot, even if they were somehow otherwise limited to a human equivalent of IQ 90.