[ben_w]

> except to the extent that dumb mistakes might result in danger

That "except" goes all the way up to starting WW3. Or a leak from a viral research lab, and by "leak" I mean "mail order" and by "research lab" I mean "the companies who already ship custom DNA and RNA retroviruses": https://duckduckgo.com/?q=companies+who+already+ship+custom+...

If you can prove that simply not training on horror stories would work, it would make a lot of people very happy.

Unfortunately, I don't think it does a single thing to solve, for example, Elon Musk just plain asking some future version of Grok to take over the world for him.

Nor would merely failing to include them in traing data stop certain entire fictional scenarios such as that Doctor Who episode where the android repair bots weren't told that the crew were off-limits as spare parts, or the other Doctor Who episode where the utilitarian robots started killing everyone who was upset because they calculated net positive utility from upset people ceasing to exist. Well, except for the bit where the Doctor saves the day, because they are not real.