[ben_w]

You're not wrong to be concerned, but you also don't need consciousness (by any of the hundred definitions that word has) to get problems.

Cancer isn't conscious[0], it doesn't hate its host body, it just optimises growth in a way that will ultimately kill its host[1].

[0] depending on the definition; IIT says everything is to some degree

[1] do/should transmissible tumours count as separate species to their hosts?

[softg]

This. LLMs don't need to be conscious to wreak havoc. The moment they reach the proficiency level of a junior programmer we're going to have a massive shitshow because so much of what we do is online and so much of that stuff is not written/maintained correctly.

It's only a matter of time before a rogue AI erases a bank's database or randomly triggers an anti aircraft missile or something similar.

[ben_w]

> It's only a matter of time before a rogue AI erases a bank's database or randomly triggers an anti aircraft missile or something similar.

Thinking of Thule early warning radar not having been programmed to know the Moon didn't have an IFF transponder and that was OK, and of stock market flash-crashes…

Both have probably already happened, though perhaps as GOFAI rather than ML.

[immibis]

Copilot is already able to do this.

To be clear, Copilot doesn't do this. Humans do this with Copilot. LLMs are still only productivity multipliers of the humans who use them, and some of those humans are very stupid.