[jacobr1]

We need to focus on the other direction. How can we have chains of trust for content creation, such as for real video. Content can be faked, but not necessarily easily faked from the same sources that make use of cryptographic signing. The attacks can sign the own work, so you'd need ways to distinguish those cases, but device level keys, organizational keys, distribution keys all can provide provenance chains that can be used by downstream systems to _better_ detect fraud, though not eliminate it.

[yetanotherloss]

The cryptography to support this has been around for ever and it's been next to impossible to make the decision makers at companies and large organizations care, much less end users.

Small time players like GE routinely fail to correctly sign industrial control software, the odds of people recording video paying enough attention to get it right and the meme crowd bothering to check even if they did seems vanishingly small without a lot of educational effort.

[jacobr1]

Yeah, you need adoption for it to work, and that in tern means there needs to be some kind of financial or regulatory incentive. But it does seem to me to be more technically feasible. Fingerprinting AI seems ... just not workable at this point.

We are starting to see adoption of software supply-chains with SBOMS, albeit imperfectly. We are starting to see increased adoption of things like DMARC in the email space to better authentic the originator of an email. Both are highly imperfect systems ... but you can start kludging something together ... and if the incentives are there I think you can build out more of a workable system.

[AnthonyMouse]

> The cryptography to support this has been around for ever

It's not the cryptography which is the problem. It's, who do you trust with the signing keys? The list inherently has to include every camera maker, despite that industry generally not having a great security culture, as well as every camera's country of origin, and every country with a security service capable of infiltrating some other country's camera maker. Which is probably all of them.

Worse, the keys have to be in the camera. Every camera. Break one of any model and you can forge images with it. Break one of any model and publish the break and you call into question every image from every camera of that type.

Then, even if a camera hasn't given up its keys, someone can use it to take a picture of a picture.

None of this requires a cryptographic break of public key signatures.

[tomrod]

> Small time players like GE routinely fail to correctly sign industrial control software, the odds of people recording video paying enough attention to get it right and the meme crowd bothering to check even if they did seems vanishingly small without a lot of educational effort.

I've wanted to build a product in this space ever since I heard about deepfakes. Mix of keybase and appropriate file hash, and hash gen for subsets of sections of video. Maybe it needs to be a protocol, maybe a product, not sure, but the need seems apparent to me.

[tornato7]

For practical uses the cryptography is quite new. Essentially if you cryptographically sign an image coming out of a sensor then you’re only getting whatever original version of it is and if you modify it even just to resize it for web for example, then it’s going to break the cryptographic chain of trust. You need to use zero knowledge proofs to allow for necessary image manipulations while keeping a signature.

[Nick87633]

At the very least an HTML link to the signed original so that people can verify the source material.

[floren]

I was thinking the other day about embedding keys in cameras, etc. but came up with the problem that you could just wire up a computer that BEHAVES like a CCD sensor and send whatever the hell you feel like in to the signing hardware, so you feed in your fake image and it gets signed by the camera as though it were real. I assume smarter people than me have put much more time into the problem, so I'd be interested to see any good resources on the subject.

[tshaddox]

I think you're essentially describing the hardware DRM supply chain.

For example, HDCP is a DRM scheme where Intel convinces (or legally requires) every manufacturer of HDMI output devices (e.g. set-top boxes, Blu-ray players) in the world to encrypt certain video streams.

Then, Intel requires manufacturers of HDMI input devices (e.g. TVs) to purchase a license key that can decrypt those video streams. This license agreement also requires the manufacturer to design their device such that the device key cannot be easily discovered and the video content cannot be easily copied.

Then, Intel gets media companies to include some extra metadata in video media like Blu-ray discs. This metadata can contain revoked device keys, so that if a TV manufacturer violates the terms of the license agreement (e.g. leaks their key, or sells a device that makes copies of video content), that manufacturer's TVs won't be able to play new content that starts including their key in the revocation list.

Of course, Intel's HDCP master key was either leaked or reverse-engineered, so anyone can generate their own valid device keys. Intel will probably sue you if you do this, I guess.

[charcircuit]

>Of course, Intel's HDCP master key was either leaked or reverse-engineered, so anyone can generate their own valid device keys

Of an older version of HDCP. New media can require a higher HDCP version where that bypass isn't possible.

[tshaddox]

That's certainly possible, but did that actually happen in practice? The downside is obviously that you essentially start selling new Blu-ray discs that don't work on any old Blu-ray players. I feel like I would have heard if that happened. Unless maybe the old players could issue firmware updates?

[charcircuit]

>That's certainly possible, but did that actually happen in practice?

Yes, see 4k blurays for requiring HDCP 2.2 requiring people to get a new bluray player.

[sebzim4500]

Interesting. I don't understand the revocation process though.

What stops the blu-ray reader from just ignoring the revocation list on the disk?

[tshaddox]

The revocation list is for the TV. Intel revokes a TV's key, distributes the updated revocation list on new Blu-ray discs, and when a compliant Blu-ray player is playing one of those new discs it will refuse to negotiate with a revoked TV.

Now that I think of it, I wonder if compliant Blu-ray players actually save the new revocation entries and then continue refusing to negotiate with revoked TVs even for old Blu-ray discs.

[galangalalgol]

If so, couldn't a malicious disc revoke all TVs?

[tshaddox]

Perhaps, although you’d presumably need Intel’s private keys in order to sign the revocation list.

[dylan604]

That's where the reversing comes in to switch the function call to check the revocation list to a NOP and just keep on going. At least, that's how I imagine HDMI equipment that ignores HDCP works

What stops them from being sold that way would probably be the licensing agreement and honest players. I'd imagine in China, there are lots of these types of devices available.

[blibble]

the blu-ray is encrypted and the player requires a key to decrypt it

if you want your blu-ray player to be able to read blu-ray disks you sign a contract that says you will respect the revocation list

if you change your mind later: your player key will be revoked and new blu-rays won't play on your device

(it's actually more sophisticated than this... they can block specific players too)

[tshaddox]

That doesn't quite sound right to me. I don't think a new Blu-ray disc could be released that continues to be readable by some old readers but is no longer readable by other old readers.

[blibble]

> I don't think a new Blu-ray disc could be released that continues to be readable by some old readers but is no longer readable by other old readers.

you can obviously think whatever you want, but you'd be completely wrong

DVD supported this 20 years ago, blu-ray's system is far more sophisticated and can even block individual players

    The approach of AACS provisions each individual player with a unique set of decryption keys which are used in a broadcast encryption scheme. This approach allows licensors to "revoke" individual players, or more specifically, the decryption keys associated with the player. Thus, if a given player's keys are compromised and published, the AACS LA can simply revoke those keys in future content, making the keys/player useless for decrypting new titles.

(from https://en.wikipedia.org/wiki/Advanced_Access_Content_System)

the spec also supports a persistent CRL so a new disk can also stop your old disks from working

[jacobr1]

I think you'd need device levels keys. You couldn't trust any particular image ... but you could perhaps know where it came from, which you gives you a better substrate upon which to infer trust.

[feralderyl]

Every time I hear about these watermarks that's what I think. There is nothing stopping someone from playing any video on a projector and simply recording the projected video on any device they want.

[mistrial9]

a very large number of consumer and high-end camera equipment does have unique ID of the device. Some metadata stores those device IDs along with other values like mfg or light settings. Its not signed into a binding chain, only marked at image creation time. The vast majority of people I expect would just copy the data files blindly.

[atestu]

What about adding other sensors like LIDAR to make it harder to fake what you’re filming? I think about it often too…

[gorlilla]

Harder only lasts temporarily until 'simple' can be commoditized. The fact we're discussing this implies there is a market for it since there is a need for any defense.

[blibble]

you'd have to do the signing inside the ccd silicon

(though if you have lots of time/effort/money you could still extract the key)

[AnthonyMouse]

Or attach the inputs on the CCD silicon to something other than the CCD array.

[Alligaturtle]

I agree with this sentiment. Years ago, I asked around at one of the smartphone companies whether it would be possible to certify to an end user that a photo is either:

1) Authentic and only lightly edited with image manipulation software (e.g., cropped, color balanced, or text placed over top of the image) 2) Produced on a phone that has had to go through hardware hacks

Note that the guarantee in (1) wouldn't prevent someone from taking a photo of a TV screen. When I asked that original question, I had quite a few more details about how the certification might be done, how the credentials would be hosted, and how the results would be shown on a website.

Anyway, just asking this question was met with a storm of negative responses. I counted two dozen messages that were either neutral (asking for clarification) or else outright hostile before the first hesitantly positive message. My favorite hostile response was that allowing people to certify images as real would steal peoples' rights. I didn't follow the logic, but the guy who made the argument was really into it.

There were lots of comments about how using AI would be a better solution, some commenting on how Cannon already did it (and messed up gloriously), others stating they didn't have faith in hardware... it makes a fella never want to ask a question again.

In the end, I got an expert to speculate that the technology currently exists, and has existed for 5-10 years, to do this with a modern smartphone. However, unless a high-level engineer or executive argues that providing this feature will somehow be a competitive advantage, there is no appetite to provide this kind of feature.

[danShumway]

> My favorite hostile response was that allowing people to certify images as real would steal peoples' rights. I didn't follow the logic, but the guy who made the argument was really into it.

My guess is likely because it seems like this would be impossible to implement without adding DRM to the smartphone and/or locking down Open Source image editors out of the attestation process. You would need to prevent access to the software, firmware, etc... otherwise the device could be virtualized or the program recompiled to circumvent the signature.

And for obvious reasons there's going to be pushback to adding that kind of DRM to smartphones. The tech does likely exist; this sounds to me like just normal attestation? It would likely hook into something like the Play Integrity API. A lot of people already hate the Play Integrity API though.

It's not the tech that's the problem, it is as you say, that people are hesitant to do it because it would require locking down the phone's software stack in a way that is widely understood by many developers and user advocates to be anti-user and in contrast to user rights to control their own devices and load their own software and/or firmware onto their devices.

I could maybe see an argument introducing some kind of signature to a raw camera input in firmware before it ever reached the user at all -- mostly just because devs seem to have given up the fight about custom firmware on a phone in general. But if you're talking about the phone signing the image after light editing like a crop has happened, at that point you're talking about moving this signature into user-space code, and while I'm sure that problem could have been explained better to you by the devs, it's not surprising to me at all that you'd get a hostile response to that suggestion because I don't see how it would be possible to do that without locking down user-space code.

[AnthonyMouse]

> mostly just because devs seem to have given up the fight about custom firmware on a phone in general

This is more like prioritization. If you can't even install your own apps, you focus on the gorilla holding a knife to your throat.

> But if you're talking about the phone signing the image after light editing like a crop has happened, at that point you're talking about moving this signature into user-space code, and while I'm sure that problem could have been explained better to you by the devs, it's not surprising to me at all that you'd get a hostile response to that suggestion because I don't see how it would be possible to do that without locking down user-space code.

That's the part that isn't a problem. If you had an existing image with an existing signature, you could modify it and store the changes as a diff against the original. You don't need or even want to sign it again, you just keep the original and its signature intact. Compressing two images that are nearly identical against each other shouldn't even have particularly high overhead.

Doing it this way would also be more secure because you wouldn't have to trust the device doing the modifications in any way.

The problem continues to be how to create such a signature to begin with, without depriving the user of control over their own property or leaving the keys inside of devices that are in the physical possession of every attacker in the world.

[danShumway]

> That's the part that isn't a problem. If you had an existing image with an existing signature, you could modify it and store the changes as a diff against the original.

I think this glosses over things a little bit. Are you going to transmit the original and the diff to every image viewer? People are talking about doing these checks on clientside devices, not just having an attestation check somewhere else.

Ultimately the only way you can check this is to give someone the original and the signature to compare. Want to blur or censor a face? Tough. Want to crop? Tough. And the person doing that verification would want to be able to look at the photograph to tell how extensive your edits were.

Technically what you're saying is true in that you could do diffs this way, but in practice you'd have to commit to publishing the pre-edit photo. We're also suddenly no longer talking about a behind-the-scenes process that just puts a little green check on the photo or something; because edits can be anything and only the original photo would be signed; so the "verification" in your image editor would now be a software stack that shows you the original photo alongside the edits I guess?

----

> The problem continues to be how to create such a signature to begin with, without depriving the user of control over their own property or leaving the keys inside of devices that are in the physical possession of every attacker in the world.

I'm quibbling though, I think we're mostly in agreement. This is the DRM aspect that people seem to be forgetting. Commentary about attestation is not making the obvious and direct comparison that controlling device behavior is already something companies are trying to do and failing at.

[AnthonyMouse]

> Ultimately the only way you can check this is to give someone the original and the signature to compare. Want to blur or censor a face? Tough. Want to crop? Tough. And the person doing that verification would want to be able to look at the photograph to tell how extensive your edits were.

Well of course they would. Otherwise what are you even trying to attest? Otherwise someone could take an image from a camera, replace literally every pixel with whatever they want and then claim it's the same image.

[danShumway]

Agreed, but at that point, why have the edits anymore; particularly if you expect people to actually check.

Obscuring faces for privacy, cropping would no longer work. And even minor touchups like lighting would be of questionable value since you're expecting users not to look at the edited photo or at least to primarily look at the edited photo next to the original.

I suspect in practice that doing edits on top of a signed photo would be basically the same as not having editing capability at all; and even that's assuming users would compare the edited and non-edited versions at all, which is not a safe assumption in my mind given how hard it is to even get people to click into a full article past the headline.

[not2b]

I could take a photo of someone else's photo with a camera that cryptographically signs the image. Then I suppose I could claim that my photo is the original (see? it is signed, with a camera that maintains a chain of trust) and the original photo is now the stolen one. To pull this off it would have to be a really high quality camera that would make an accurate copy.

Perhaps something like this is what your hostile responder was thinking of.

[tornato7]

But if both images had a timestamp in the signature, wouldn’t you be able to prove that the original was taken first?

[AnthonyMouse]

How does the device know what time it is?

What happens if the original was taken with any existing device that doesn't make signatures, or is a model that subsequently had its keys revoked?

[tornato7]

You can store the hash on a public Blockchain and the timestamp of that transaction will be verifiable.

[AnthonyMouse]

If everybody does that with everything, blockchains get infeasibly large. If not, anybody can go and register things on a blockchain that the original creator didn't and then claim they were first.

[nofunphil]

Agreed. At the risk of a shitstorm of downvotes, tokenized media could be part of a solution, especially at the consumer level. Authenticate real videos via a mint button/QR that airdrops you a token from creator. May require platforms to opt-in tho. Basically trust nothing unless you can authenticate source onchain. Not great fo sho, but prob necessary soon

[tornato7]

This is the best solution IMO - Blockchain allows others to independently verify the time that media was created without needing to trust some random company's database / API in perpetuity.

[nonameiguess]

What's the threat vector you're trying to mitigate here? If you're wondering whether a movie that claims to be produced by Disney really was, if it's in theaters or on Disney+, then you can trust it was actually made by Disney or at least licensed to them. As long as the Washington Post still employs its own photographers and doesn't accept imagery submission from the general public, you should be able to trust a photo published in the Washington Post is at least of something real, unless you just don't trust the Post itself. If you're thinking a YouTube channel or something, unless the channel got hacked, seemingly anything published there was really published by the channel owner. Maybe they're showing you something made by AI that isn't real, but as the owner of their own signing key, nothing would prevent them from signing an AI-generated image.

If you're talking someone on Twitter or Facebook is putting a photo in your feed claiming a human photographed BLM throwing bricks through a window, don't trust shit being posted on Facebook or Twitter no matter what, probably, but even there, unless the profile was hacked, you either trust the person who owns it or you don't. Nothing would prevent them from signing a forgery of reality that they legitimately forged themselves. Even with device-level keys, what are you trying to prove? You can pay actors to throw bricks through windows.

I guess the concern is this doesn't scale as well as asking Midjourney to do it, but I wonder to what extent that is even true. With 8 billion people on the planet and counting and a whole lot of them doing this shit, given the limited input bandwidth of human sense organs, there is seemingly some maximum saturation of bullshit a person can be exposed to that a lot of people have already hit, and having the Internet host even more of it doesn't mean they'll grow bigger eyes and a faster brain that can actually ingest more bullshit than it already does.

[atestu]

Well, a lot of people don't trust the Washington Post and might want proof that the photos they're posting are of real events.

The Washington Post might not trust its photographers completely either (journalists making stuff up happens[0]), so they too might want proof the photos they're getting are real.

[0]: https://www.nytimes.com/2003/05/11/us/correcting-the-record-...

[danShumway]

I don't know; are you imagining a world where the people who believe the Washington Post would purposefully fabricate photos will be convinced by running a signature check on those photos?

Also bear in mind that Photoshop does exist today, and even without AI it would be well within the budget of the Washington Post to do edits to photographs that would be good enough to convince most non-experts. I don't run into many people saying their photos are doctored, but the conspiracy theories around photos today don't seem to be swayed by saying, "experts say it's not doctored."

I think if a chain of trust requires ordinary people to check signatures, it's probably not going to matter very much? I've seen people comparing this to HTTPS; people don't check HTTPS certificates either. There's a reason why PGP and signed messages haven't taken off. And the biggest criticism of Matrix that I see today when I introduce new users is that identity verification is too hard and they don't care about it. And these are the reasonable people.

I'm trying to imagine talking to someone who believes the mainstream media is lying about everything and telling them that, "no, it's OK, Google the tech company that you hate and that you think is trying to swing the election checked some math from its hardware and it says the photo is real." I don't think that's going to persuade those people of anything.

[jacobr1]

Well, if I see a clip on youtube claiming to be from Disney ... it really might not be.

If I see a photo on twitter claiming to come from the Washington Post, it might not be.

If I see a photo in my facebook feed of a rioter, did it come from poster, or are they just reposting something else? Did that repost come from a newsource I trust, like the WP in this case, or from some reddit post, maybe edited or synthetically generated?

> Maybe they're showing you something made by AI that isn't real, but as the owner of their own signing key, nothing would prevent them from signing an AI-generated image

That's right. This only helps narrow the source down, then you still need to decide if you trust the originator. But I think a lot of the problems we've seen with social media disinformation is the wide dispersion of content claiming trustworthiness from a reputable source, falsely.

[danShumway]

All of those are current problems though. Signing media isn't hard, determining who is trustworthy and how to do identity is hard. This the HTTPS problem; you can show that the connection is secure (assuming people care enough to check in the first place), but you still have to check if the connection isn't actually to Target and is to Torget instead.

Signing only helps if we have a reliable way to verify identities out of band, and we don't, that's one of the reasons the other problems you mention still exist today. Today, how do you determine that an article is actually from the Washington Post? You check the website. It's not like verification of whether an article exists is hard today -- it's everything else around it that's hard.

If checking if an article is actually on a website is too hard for people, importing a signing key is also going to be too hard. If people are confused remembering the Washington Post's URL, they won't learn how to use a signing key to check identity. Now maybe a website could automate that and put some kind of verification badge next to trustworthy verified identities, but I'm skeptical because Facebook did try to do that with news sources and it was a disaster and a bunch of politicians accused them of censorship.

We have a lot of mechanisms for verifying identity and sources of information that aren't leveraged today, and I think the immediate question to ask about a chain of trust is "what's going to make this different from all of the other chains of trust that people are already ignoring today?"

[ryukoposting]

I was thinking the same thing- we've had digital signing algorithms for decades, and those seem to work fine enough. There's a healthy distrust of cert authorities, sure, but it still works.

[p1necone]

The signature only proves that the website/Bill the photographer was involved in the chain sending the website/photograph to you, not anything about the content itself.

Unless you have reason to trust Bill himself you can't trust that he actually took the photo, or that it isn't ai generated. Although knowing that Bill isn't tech savvy enough to do those things might be enough.

[jacobr1]

That ... might be all we get though. If anybody can produce artificial, but realistic content that is indistinguishable from something the real thing, then all we might have is our willingness to trust any given originator/distributor of the content.

Narrowing down the problem to "Do we trust Bill" is at least something we can attempt to address. This also eliminates some sources where we can more easily discount things. If the source is 4chan, maybe I don't need much more information to make my repetitional assessment.

[jallmann]

Agreed that cryptographic provenance seems like the best way forward, however the goal is more about proving authenticity, less about detecting fraud. Small but important distinction.

There are industry initiatives around this already such as CAI https://en.m.wikipedia.org/wiki/Content_Authenticity_Initiat...

[jacobr1]

CAI is roughly the kind of scheme I was suggesting. Thanks for the reference.

My take is that proving authenticity might not be something we can do with any degree of accuracy in a general sense. So if that is infeasible, then we need _some_ kind of mitigation. Something like CAI allows us to make the an assessment about the how much trust to give an informational source, probably taking into account multiple factors (known exploits in the source device, reputation of originator and what claims are be attached in the metadata). This might allow me to accept that a given video originated from a local tv station, rather than tiktoker edit, but I still need to asses if that station used genAI or has been compromised or whatever else. But that seems a much narrower reputational problem, that also will be contextual.

[jallmann]

Exactly - content authenticity is a narrower and more tractable problem.

And I suspect in most cases that is enough - the world runs on trust, so if a reliable source (eg, the NY Times) attests that a photograph is authentic, then we can reasonably trust that. And as you noted, this chain of provenance can go all the way back to the device itself.

Fraud, on the other hand, is much harder to prove without a doubt. It is still a problem, but probably less so in the general case [1]. The concerning thing seems to be extremely targeted attacks, eg, hacking a CCTV to implicate someone in a crime.

[1] Notwithstanding folks who slurp up unfiltered content on, eg, TikTok, but the most outrageous stuff usually doesn't "make the news" on its own without more vetting. If anything, this will reinforce the importance of actual journalism, fact-checking and corroborating evidence.

[tempusalaria]

You can see here: GitHub.com/HNx1/IdentityLM

It’s a direct (and open source) implementation of public key cryptography into the LLM logit distribution.

The paraphrasing model/beam search needs work - feel free to pitch in :)

[j0hnyl]

The problem is that this requires a federated digital identity provider which is kind of the antithesis of a utopian internet.

[flextheruler]

Would it really be that different than https? I would assume it would mainly be large entities like governments or corporations utilizing these certificate chains

[AnthonyMouse]

The certificates for HTTPS certify that you're communicating with someone the CA verified has control over some domain name. They don't require the CA to exercise any control over the endpoint device or for the user to cede any.

If you want the camera sensor in your phone to certify that the user hasn't altered the image, the device can't be in the control of the user.

[tudorw]

"Magnetic anomalies are generally a small fraction of the magnetic field. The total field ranges from 25,000 to 65,000 nanoteslas (nT). To measure anomalies, magnetometers need a sensitivity of 10 nT or less."

Would signing content with a cryptographically consistent encoding of this field be workable?