[jlokier]

What an interesting idea!

Several commenters point out end-to-end encryption would prevent filtering or tagging messages.

But that's not true. The message analysis could be done at either endpoint without violating privacy.

Tagging (or removing) a message before you send/forward it, or after you receive it, with "the central message of this comment has been tagged as "probably a hoax" by hoaxtracker.com; check out this CDC notice <here> to learn more".

<here> does not need to be a URL which reveals much other than your general interest in the subject. But if that seems too revealing, it could already be already available as part of the endpoint's filtering data and readable locally.

Lots of people forward (retweet), or write a little something before resharing what is false or misleading information, not realising they're doing so. I would not be surprised if getting those tags, rarely enough to stand out, before they send the message would cause some people to hesitate and check/think a bit more before sending. Maybe rephrase their attached comment into a question rather than confident outrage.

Technically this is not much different from privacy-preserving spam filtering.

[imagiko]

Whoa wait what..is it really E2E if they can analyse the content before putting it through their network?

[panda88888]

Yes, they can analyze it in the app on user devices before it is encrypted and transmitted. The app on user decide need access to clear text in order to encrypt the message. Same on the receiving end. The app on user device can analyze the message once it is decrypted on user device.

[kshacker]

Wow did I not read something similar last couple of days about on-device machine learning being better than in the cloud machine learning and how Apple got that right. That's where this may also be going then if we follow your train of thought.

[jlokier]

On device is where you want it if it's going to analyse really private data, or something effectively your own (such as homomorphic encryption, or a link to your own computers elsewhere).

You're likely to feel so much happier, freer and easier sharing your most personal life datastream with an AI assistant, if you can be sure its most intimate analysis is just between the two of you.

It puts me in mind of the two kinds of AI in Manna: https://en.wikipedia.org/wiki/Manna_(novel)

Coincidence or not, the dystopian AIs are somewhere in the cloud and work for someone else, while the utopian AIs are intimately personal to each user and work just for the user.

[throwaway287391]

Sure why not? As long as no data ever leaves the device unencrypted and the encrypted data can only be decrypted by the client at the other end. Of course you'd probably have to take the app's word for it that that's actually what it's doing if you don't have the source, but that's no different from current E2E encryption offerings from WhatsApp etc.

The part I'm not sure about is whether the on-device certification that the message is "clean" couldn't be (easily) spoofed. But it would probably help curb distribution of illegal material anyway.

[thu2111]

No, obviously not. The mental gymnastics involved here are impressive: the point of E2E encryption is to stop the service provider seeing or tampering with your messages. If they do that anyway it doesn't really matter how it's implemented. They could also just use a broken random number generator, or many other ways to implement the policies whilst still having encryption code in the product. It's the end result that matters, not the precise means of implementing it.

[imagiko]

Phew, agreed. I mean of course the company "can" read the message. If it does, I would love to see that shown by the app upfront, so I can avoid using it.

[api]

Analysis happens on either end, not the network or servers. Of course if both ends are "cracked" this doesn't work, but the goal is to stop mass spread of disinformation. Most people won't modify their client.

[jpsalm]

Of course the party that is doing the encryption can see the original content...

[paganel]

> But that's not true. The message analysis could be done at either endpoint without violating privacy.

This is stupid. Lots of naked baby photos get sent in my culture (Eastern-European country) in a most non-harmful way, i.e. from parents to the kids' grand-parents or even to the parents' close friends (especially from the mother to her friends). Your supposed filter will most probably block that social-sharing process (because it will see photos of naked children => very, very bad), not realising the above mentioned context.

[jlokier]

> This is stupid.

Truth is truth. Saying something is possible isn't the same as advocating that it be done, and it's useful to point out something is possible when people at first seem to think it is not.

Also, I was talking about messages, not baby photos, and with regard to misleading or false information, hoaxes etc that cause people to behave more dangerously to others during a pandemic. Saving lives, that sort of thing.

If it's giving the user advice that others have judged what they are retweeting to be a hoax or bad medical advice, that's not blocking, it's providing context. If they don't like it, they should be able to dial it down.

With regard to baby photos, if a network starts blocking those due to poor filtering, I would hope people switch over to another network that lets them share the photos.

> Your supposed filter will most probably block

I wasn't talking about filtering particularly, the emphasis was on providing a note to the user. Much like when Twitter attached a note to Trump's tweets.

In any case, the analysis I had in mind is not "skin tone filters" and that sort of nonsense. It's not meant to be thought police, working for someone else. It's meant to advise the users themselves to think again about some content. At least at the currently level of sophistication, that would be "we recognise this particular message or photo".

There are better and worse ways to implement it of course.

[benlivengood]

The problem is that spammers/shills would have full access to local filters and could tweak their messages to trivially bypass them.

[user9909]

This is a major privacy violation. Who would willingly install malware that censored their posts? I've never seen a generation beg for LESS rights.

[jlokier]

Something that reveals no information to others is not a privacy violation by definition.

(Although, something that blocks communication (which I don't think I agree with anyway) based on local analysis is an autonomy violation. But not a privacy violation.)

Something that tells you when you've just received a well documented hoax is not malware, it's probably useful, and most people will probably keep it switched on if the quality is consistently good.

By your logic, spam filtering (outgoing and incoming) is also malware, and a privacy violation. (Even though it protects people against malware, and indirectly protects privacy.)

Do you believe spam filtering is bad? I doubt it.

Yes, people ask for certain kinds of anlysis based message blocking all the time. We begged and pleaded for better spam filtering 20 years ago because the vast majority of messages were pure spam and it had made email difficult to use. It's a major reason people switched to Gmail, because other providers' spam filters weren't good enough.

I think the key feature most people would want in any kind of alerting, tagging or filtering is that it does what they want, rather than what the enemy wants, as it were. As people's preferences differ, that can only happen if it's configurable by them rather than blanket imposed. Things like ad blockers work this way - you can change the defaults if you want - and people seem to like those.