An LLM is a large and complex machine, not a screwdriver. Large and complex [physical] machines are built with safeguards to prevent misuse, injury, etc by regulation.
LLM's are in principle text in / text out machines. If the user extends its capability to have agency over a production database or a machine, there's nothing that can safeguard the safety.
Imagine I ask an LLM to instruct left/right/speed up/slow down while driving. I can simply bypass any safeguard by stating i suddenly became blind while driving a car. While in fact i'm blindfolded and doing an experiment on a highway.
A bulldozer is a large and complex physical machine, yet it has (almost¹) no safeguards against misuse or injury. It's all operator training. Lathes tend to not have doors/enclosures, in particular large ones. You get taught where to not put your fingers, and to wear safety goggles. Cranes don't have a lot of safeguards either, you better know how to attach things; hardhats aren't gonna do sh*t if you get a ton of concrete dropped on you.
etc. pp.
I'm not sure where this "tools are made to be safe" belief comes from. This is only the case in "consumer" environments. Of course you don't intentionally make things unnecessarily unsafe, but — in a professional environment there is an expectiation that the operator had training and knows what they're doing.
Maybe that's what we're missing: training in safe AI use. With a certificate that has to be periodically renewed. At the current rate things are going, I'd say 3 months is a good renewal cycle ;D. </s>
(¹ it beeps when it goes backwards. Honestly, I'm not sure that counts for much.)
Imagine I ask an LLM to instruct left/right/speed up/slow down while driving. I can simply bypass any safeguard by stating i suddenly became blind while driving a car. While in fact i'm blindfolded and doing an experiment on a highway.