[hn_throwaway_99]

> We strictly moderate the generated text of any sexual, offensive, racist, or threatening content.

This is exactly what makes me so angry about "AI safety" initiatives: they are largely worrying about the wrong thing. People have been so focused on the "this may make some obscene joke, or be biased against some skin colors" that they have completely missed out on the much more serious harms that AI will cause with respect to, in this case, impersonation scams.

Congrats, people can't say the N-word with your technology, but they can say "Hi Bob, just calling to verify that we did indeed change the target account where you should wire your invoice payment."