Hacker News new | ask | show | jobs
by ben_w 744 days ago
What would you need to see, short of the actual end of the world, to take the risk seriously?

I don't have to be in a car crash or get shot in the head to know this is a bad thing, and nobody sane is going to bother causing the end of the world just to convince you it's possible.
1 comments
c1sc0 744 days ago
Maybe a small-scale demonstration then? Seems to have worked so far for nuclear weapons proliferation.
link
ben_w 743 days ago
> Maybe a small-scale demonstration then?

What, precisely, does that even mean?

Critics (I initially said "you", but rereading this is ambiguous) clearly don't accept anything that currently exists as such a demonstration: not the models which are superhuman at strategy games; not the automation actually used by the real militaries despite dangerous flaws (whose bugs have resulted in NATO early warning systems being triggered by the moon and Soviet ones by the sun, or planes nose-diving because of numerical underflow); not the use of LLMs to automate propaganda; not Cambrige Analytica; not the lack of controls that resulted in the UN determining that Facebook bore some responsibility for the (ongoing) genocide in Myanmar; not the examples given in the safety report on GPT-4 prior to release showing how it was totally willing to explain how to make chemical weapons; not the report the other year where a drug safety system was turned into a chemical weapon discovery tool by deliberately flipping the sign of the reward function; and not the OpenAI report on maximal misalignment in their own models caused by flipping the sign of a reward function by accident.

What is the smallest "small scale" demonstration that people who currently laugh at the idea of the possibility of a problem, won't ignore?

link
c1sc0 743 days ago
Flipping the sign on a reward function to protect us from the next pandemic should fit the bill. If we ignore that I guess we deserve it?
link
ben_w 743 days ago
> Flipping the sign on a reward function to protect us from the next pandemic should fit the bill

I don't understand, are you suggesting flipping the reward function of reproductive fitness itself, in vivo, of DNA/RNA?

And how is "protect" supposed to demonstrate danger? That's like saying "ACAB protestors are dumb, I'll only believe the police are evil when they catch a gunman"?

link