Hacker News new | ask | show | jobs
by nieve 2418 days ago
A false positive rate of 1/1000 is hard to assess without actual prevalence stats, but with a decent-sized userbase it seems likely you're still going to get a significant number of false positives. Is it intended that users of your system would have employees manually vet all positives (with legal and mental health concerns) or just submit them without review? I'm coming from having built tools to support a large manual sweep in the 2000s and watching the toll it took on my coworkers.
2 comments
faustomorales 2418 days ago
Great question — organizations decide how to handle reviews internally. So the answer to your question on “review all” versus “automatically submit” is a, perhaps unsatisfying, but honest: it depends. We provide a guide [1] to help organizations formulate their own policies. And we're currently working on a content moderation tool that focuses on helping organizations operationally handle problematic content and considers the wellness and resiliency of reviewers.

[1] https://www.thorn.org/sound-practices-guide-stopping-child-a...

link
nieve 2418 days ago
Oh good, I'm sure most organizations can use something like that guide as well as the tools. There's a lot of legitimate worry about both the wellness side & the legal exposure issues, but it seems like beyond the common wisdom to be very careful (somehow) I think in a lot of minds there's a lack of clarity as to what exactly that means. Is there a particular reason access to the guide requires handing over contact information?
link
mayniac 2418 days ago
>I'm coming from having built tools to support a large manual sweep in the 2000s

Any chance you're one of the devs behind C4All?

link
nieve 2417 days ago
No, thank god. I was hired as technical lead for the team put together when my new employers inherited a medium-sized low profile social network primarily popular in SA and SEA from their parent company. We got a report on something in the supposedly mostly unused photo sharing feature and discovered there were no moderation tools at all, so the first thing I had to do was build something to even verify the reports. At that point it turned out to be necessary to do a sweep and legal thought it should be done by hand, so my coworkers spent days going through it. It would have been grueling for them even if they hadn't run into the awful material.

(I escaped having to help because my sole minion was the kind of guy who decompresses a several gb bzipped log file as root in _/root_ and wanders away while it's running.)

link