[BingoAhoy]

"Big tech permabanned me and doesn't mind false positives"

Isn't this often the case with humans minus the algorithm making the decision? Many times of the few times I've been in trouble, with HR, the law, or whatever authority you realize doing things that seem bad but aren't actually bad is almost as dangerous as doing something actually harmful because turns out humans aren't very good judges of ambiguous cases in low information environments.

Even when not ambiguous humans by and large don't have a good grasp of what is or isn't moral. And they typically show a large lack of empathic ability for how their actions will effect others.

[PoignardAzur]

Yeah, I'd say "whether humans are involved" isn't actually a good metric for whether abuse detection is fair / avoids screwing you over.

A better proxy would be "how much the company spends per user to detect false positives". Whether it's human oversight for each case, or engineering time spent fine-tuning algorithms to exclude known false positives, the more the company spends, the less it's going to screw you over.

(In practice, companies want to spend very little, which is why you get underpaid Mechanical-Turk workers and slapped-together detection systems.)