[LukaAl]

As other has pointed out, this is extremely creepy. The easiest solution would have been this one:

- The user uses a JS solution to hash the images on the client, without the image being uploaded

- She compiles a form with additional information (e.g: capture her account, reasons for uploading, suspect person sharing the picture)

- The picture is saved in the DB as un-verified revenge porn.

- The first time someone uploads a picture that matches the hash, the pic is quarantined and the specially trained individual manually check them

- A scoring system could be used to check the reliability of the submission. If multiple photos marked revenge porn get rejected, the control becomes ex-post. For even more violations, the user get banned from using the tool and should directly contact Facebook. Submitting the same hash that has been rejected, will count as a "red mark"

Now, I understand this system is very complex, what Facebook has done is an MVP and as a product manager, this is what I prefer. But considering the issue (revenge porn, not something I necessarily want to test the impact on retention :-) ). Also, yes, it requires resources, but Facebook has a problem with trust lately, better to do the best...

[edited for formatting]

[evgen]

Now figure out a way to do step #1 (user does a hash client-side) without making it trivial for someone else to create a filter that adds enough noise to invalidate step #4 (uploaded pics that match a hash are quarantined.)

[wjh_]

PhotoDNA would be an option, I imagine. Or at least something similar!

https://en.wikipedia.org/wiki/PhotoDNA

[moyix]

Honestly, I doubt that most of these sorts of algorithms would survive concerted attacks – that's why they tend to be closely guarded.

Alex Stamos (Facebook's CISO) implies this is why they can't do it client-side:

https://twitter.com/alexstamos/status/928646228472078336

[dannyw]

The attacker isn’t a national state adversary. The attacker is a vengeful ex. The chances of such attacker even been aware of what hashing is, is close to zero.

Whoever designed this system is probably heavy on security, but low on product.

[kelnos]

No, the attacker is a bored hacker who wants to create problems for FB, just because they feel like it.

[lozenge]

And when the attacker Googles "how to upload images banned by Facebook" won't they find that article from that one vengeful ex who understood the technology well enough to build a tool?

[LukaAl]

From when security through obscurity is a good idea?

[syrrim]

This is in general an unsolvable problem, and the only possible security is through obscurity. Images are on a spectrum, and so in order to hash an image, you must quantize the spectrum. Between two images, there is a continuum of mixed images. Using bisection, you can identify the point where it goes from matching to not matching. You can then upload the non-matching variant. Because of the way hashing works, this will be indistinguishable from any other non-matching image. Therefore, this creates an easy method to bypass facebook's filter while sacrificing minimal quality. As long as facebook matches against hashes, there is way to prevent bypassing of this form. All they can do is make it as difficult as possible.

[_red]

You should go put a $100 bill on the dashboard of your car along with your laptop on the front seat. Report back the results.

snark-mode off: We use obscurity every day and its a completely valid layer of security.

[LukaAl]

> We use obscurity every day and its a completely valid layer of security.

Not sure I agree with that, most of the time when we do that is because we don't want to spend the time to have better security. And then we get burned. To your example of the 100$ bill: at my parents home with the car parked in the garage? No problem to do that at all. Out on the street in SF? No. I don't trust my glass enough as a security measure. But I don't leave money at all, is not security through obscurity.

But we are going OT. The problem that is raised is that they need necessarily security through obscurity. And we have two problems:

- How really robust are these algorithms? How long before we will see people abusing them?

- Have you thought hard enough about how this system could work? E.g: have a partial hashing made client-side and the final one on the server? Or a situation where the server code is open-sourced without the model to calculate the hash? That would allow for external review without disclosing the hash. Yes, you still need warranties that Facebook is using that code, but you could have a trusted third party certifying the program.

My point is, the person who designed this program didn't really understand the problem. The problem is not revenge porn. The problem is Facebook reputation. And this solution is totally deaf.

[dannyw]

Why would you need to? We are talking about vengeful ex-es here. It’s possible to bypass this trivially by uploading it to another image sharing service, so the goal should be defection in the general case; not try and protect against the 0.1% that even know what a perceptual hash is.

[LukaAl]

Agree, that's a problem but there are options to solve it. Look at PhotoDNA by Microsoft [0]. But it is a second step. First, you need the reporting properly done.

[0] https://www.microsoft.com/en-us/photodna

[agumonkey]

Good point, that said, with the recent visual ML trend, I'm sure fb or the likes can engineer a solution for that.

[tree_of_item]

How does giving Facebook the image solve that problem, in a way that can't be done client side?

[jsjohnst]

This is already a solved problem at FB (PhotoDNA does this for them for CP images).

[the8472]

> - The user uses a JS solution to hash the images on the client, without the image being uploaded

You have to trust facebook in either case, each time you do it. Either to handle your nude pics properly or to serve you javascript that does what they claim it does, every single time.

On the other hand an open source desktop application only needs to be audited once and then can be validated based on a hash.

In browser crypto is not a solution if you want to minimize the needed trust.

[oh_sigh]

Nobody wants to run a desktop application given to them from facebook

[hnzix]

But it could force autoinstall from the website and autoscan your drives then preupload your nudie pics and tag them with your name. How convenient! If all your friends are using it then it must be safe right?

[the8472]

> only needs to be audited once and then can be validated based on a hash.

I thought I covered that concern, but I neglected to mention that it should be open source so everyone can audit it.

[statictype]

It’s better than sending them NSFW photos for human review

[zodiac]

You could examine network traffic to confirm that not enough data is sent to reconstitute the picture

[the8472]

They could save it indexedb or localstorage and send it later. So now you have three things to inspect. They could even shunt it to a different domain within the browser through iframes and messaging. Now you have another thing to watch. There aren't even any devtools to watch that.

The cheese is full of holes.

[icebraining]

You'd have to check every single time, since the code can change at any time.