[kylestanfield]

Realistic AI video generation will put us firmly in a post-truth era. If those videos were 25% more realistic it would be indistinguishable from a real interview.

The speed with which you’ll be able to create and disseminate propaganda is mind blowing. Imagine what a 3 letter agency could do with their level of resources.

[a_wild_dandan]

Content authenticity follows from the source's credibility. This is why chains of evidence are crucial. If squinting at pixels were the harbinger of a post-truth reality, then it's already many decades late.

[Gigachad]

That's what we have now where people flat out refuse to believe something unless it's being reported by their news source of choice. Only the sources people trust as true aren't aligned with actually being true.

[pkoird]

Good luck telling that to aging grandparents and people in third world countries who are just getting acquaintanted with the internet.

[dingnuts]

even in first world countries grandparents are reading the print version of The Epoch Times lol

[JumpCrisscross]

> speed with which you’ll be able to create and disseminate propaganda is mind blowing

The problem isn’t the fakery, it’s this speed of dissemination on algorithmic social media. It’s increasingly looking like the modern West’s Roman lead pipes.

[andrepd]

Yet people in this very website are 200% hyped about GenAI because it makes it easier to generate slop frontend code or whatever.

[9283409232]

People on this website are externalized from the consequences of their actions or make blog posts about how they went on a digital detox retreat, joined a cult, found themselves, left the cult, and founded a new startup that will not be evil until it needs VC money then joins YC and starts being evil.

[baxtr]

Text can be mass faked since Guttenberg, photograph since at least decades.

What makes you think fake videos will have an outsized impact?

[ofjcihen]

Text - “we have no idea if they actually said that”

Picture - “this could be out of context” (this is used constantly in politics and people fall for it anyway)

Video removes the question of context and if the person actually did it. So now instead of printing about it or showing an awkward picture from a certain unflattering angle I can generate a video of your favorite politician taking upskirt photos on a city bus.

As the tech gets more and more realistic we’re increasingly straining the average persons ability to maintain presence of mind and ask questions.

[baxtr]

Yes! And:

Video - “There’s no way to tell whether this is AI-faked or not.”

[ofjcihen]

Ha, I’d love to share the optimism that that’s what most people would say but what we’ve seen is the opposite. The more convincing the media (as in media format) is the more people are willing to believe it.

Don’t get me wrong. I hope this level of fake media causes people to stop taking things at face value and dig into the facts but unfortunately it seems we’re just getting worse at it.

[heylook]

A picture is worth a thousand words.

[phkahler]

>> Realistic AI video generation will put us firmly in a post-truth era.

Verifiable authenticity might just be the next big thing.

[eloisius]

It’s already a thing in some cameras. New Leicas will embed a cryptographic signature in the photo.

[madaxe_again]

People don’t want authenticity. They want confirmation of what they already think.

[wizzwizz4]

People want authentic confirmation of what they already think, which makes it easy to give them the confirmation and lie to them about the authenticity.

If they can't have both, some people prefer confirmation to authenticity. But that's far from a universal preference.

[drdaeman]

I think you're over-dramatizing this. It's not "post-truth" - nothing about things being true or false changes. Or at least when I'm reading "post-truth" I'm thinking about a world where truth is unknowable as it's indistinguishable from a forgery. That's not what's really happening, though.

I would like to suggest a different lens to look at it through. In my understanding, we merely had a very brief period of time where a probability of mass audiovisual media from an untrusted source being an inaccurate portrayal of events was low enough we very rarely bothered to actively think about it and consider if it could be "fake". Or I would rather say "artist depiction", to make it match pre-mass-photography world more accurately.

All that changes is this probability. It used to take a lot of resources to make a good realistic-looking media, now it's becoming a commodity. What required a resources available only to a few, is now rapidly becoming cheap. That's really all what's happening - the rest is all the same, our senses can still lie, and we can still believe in things that never objectively took place.

And it's only about the realistic-looking audiovisual media. Other types of media (such as texts or non-realistic graphics) had those issues since forever. And we're still mostly fine (though the rapidly increasing scale of mass communications poses their own challenges, of course, but that's a related but different issue).

There are at least two ways I beleve this will be approached:

1. Updating mental models of trust, so we don't eagerly believe random people posting videos on the Internet anymore. Not easy (especially for old timers, as we are tend to be quite firmly set in our ways of thinking), but next generations would be born to the new reality and misplaced trust won't be an issue again. The obvious danger is the instability during the transitional period - societies have huge inertia, so a few decades could get quite messy.

2. Attempt to legally or technologically keep said probability low. Trusted cameras, AI regulations, extra online policing - those kind of things. The obvious dangers are that this could have all sorts of weird implications for freedom of speech, personal computing accessibility, and could also contribute to increasing power disparity.

Whatever happens in the future is unknown. All we can do it to try our best to steer it in direction we think is right.

[dyauspitr]

To be honest I can consider myself pretty savvy when it comes to identifying fakes and the schoolboy ones would have fooled me. Pretty flawless videos and the accent and lip sync were spot on. I don’t even think you truly need that extra 25% for the casual observer.

[rightbyte]

I don't agree. 3 letter agencies have been able to fake videos since the inception of videos. Even more so with CGI.

It has always been about trust in the authors.

The main difference is petty fakes would be cheap. I.e. my wife could be shown a fake portraying me for whatever malicious reasons.

[heylook]

You're being obtuse. There's an obvious difference between "state-level actors can produce misleading films" and "anyone with an internet connection and 5 minutes can make anything they want".

[krapp]

Not in terms of effect. This might have been a gamechanger 20 years ago but nowadays people already trust TikTok memes more than they trust CNN. The bar for credibility is so low that this sort of thing is almost trying too hard.

[ofjcihen]

“People” aren’t a monolith. Certain people are definitely falling for low effort TikTok trash but now more people will fall for these more “credible” fakes.

[rightbyte]

I think it will be like these "X celebrity is dead" fake articles that went viral on Facebook 201X something. People, as in enough people to make gossip, will only get fooled 3 or 4 times.

[rightbyte]

The post I responded to wrote "Imagine what a 3 letter agency could do with their level of resources" and I don't think much changed in that regard.

[hcarvalhoalves]

The Philippines has a story of foreign influence on their local politics. It wouldn't be crazy to expect this is just the latest chapter of the 3 letter using it as laboratory.

[patrakov]

No, it won't.

Expected reaction: every camera manufacturer will embed chips that hold a private key used to sign and/or watermark photos and videos, thus attesting that the raw footage came from a real camera.

Now it only remains to solve the analog hole problem.

[ghushn3]

I don't think that's as trivial and invulnerable as you think it is. You are talking about a key that exists on a device, but cannot be extracted from the device. It can be used to sign a high volume of data in a unique way such that the signature cannot be transferred to another video.

Now you have another problem -- a signature is unique to a device, presumably, so you've got the "I can authenticate that this camera took this photo" problem, which is great if you are validating a press agency's photographs but TERRIBLE if you are covering a protest or human rights violation.

[hansvm]

- Attach a screen to the camera. Bonus points for bothering to calibrate that contraption.

- Watermarking is nearly useless as a way of conveying that information, either visibly distorting the image or being sensitive to all manner of normal alterations like cropping, lightness adjustments, and screenshotting.

- New file formats are hard to push to a wide enough audience for this to have the desired effect. If half the real images you see aren't signed, ignoring the signature becomes second-nature.

- Hardware keys can always be extracted in O(N) for an N-bit key. The constant factor is large, but not enough to deter a well-funded adversary. The ability to convincingly fake, e.g., video proof you weren't at a crime scene would be valuable in a hurry. I don't know the limits, but it's more than the 2-10 million dollars you need to extract a key.

- You mentioned the analog hole problem, but that's also very real. If the sensor is engineered as a separate unit, it's trivial to sign whatever data you want. That's hard to work around because camera sensors are big and crude, so integration a non-removable crypto enclave onto one is already a nontrivial engineering challenge.

- If this doesn't function something like TLS with certificate transparency logs and chains of trust then one compromised key from any manufacturer kills the whole thing. Would the US even trust Chinese-signed images? Vice versa? The government you obey has a lot of power to steal that secret without the outside world knowing.

- Even if you do have CT logs and trust the company publishing to them to not publish compromised certs, a breach is much worse than for something like TLS. People's devices are effectively just bricked (going back to that 3rd point -- if all the images you personally take aren't appropriately signed, will a lack of signing seem like a big deal?). If you can update the secure enclave then an adversary can too, and if updating tries to protect itself by, e.g., only sending signed bytecode then you still have the problem that the upstream key is (potentially) compromised.

- Everyone's current devices are immediately obsolete, which will kill adoption. If you grandfather the idea in, there's still a period of years where people get used to real images not being signed, and you still have a ton of wasted money and resources that'll get pushback.

Etc. It's really not an easy problem.

[AlotOfReading]

Persistent media watermarking through the analog hole is a solved problem and has been for years. It's standard practice on films.

What does it even mean that hardware keys are extractable in O(N) time? If there's some reasonable multiple of N where you can figure out a key, your cryptosystem is broken, physical or not.

It's also very straightforward to attach metadata to media and wouldn't take a format change.

[card_zero]

The problem would be spurious watermarks, not vanishing ones. Create fake video, point camera at screen, re-record it. Now it's fake and authenticated as a genuine camera recording.

[hansvm]

> cryptosystem is broken

The problem isn't the cryptography. It's the hard part of cryptography, key management. In this case, the key is in the hands of the attacker, written down for all to see, and the only real defense is that it's written in a tiny "font" next to a bunch of other data.

You're right, that's a broken model. That's exactly what I'm saying. It's kind of like the current craze of using a phone number on login for "security" -- doing so increases the cost to an attacker by a fixed amount ($10-$15 or so) and is only reasonable for security (as opposed to selling out to data brokers) if the value to the attacker of the account in question is less than that. The security doesn't come from actual impossibility guarantees but by trying to stack a big enough constant factor onto the thing that nobody actually cares to break it.

> metadata vs file format change

Ah, exif and all the other slightly underspecified and incompatible metadata formats that no vendors fully agree on and are stripped by various middlemen and not relevant when, e.g., iphones auto-lossily-compress the formerly signed images when emailing and whatnot. We _could_ use those, but I don't think the software work involved to actually make that technosocial system work correctly is less than a new file format (e.g., with progressive rendering and something conceptually equivalent to signing each progression to account for compression use-cases).

> persistent media watermarking

Last I checked, failure rates, both type I and type II, are in the single-digit percentage ranges or worse, and that's with very low bit rates. Is the SOTA much better than that now?

[AlotOfReading]

Physical extraction doesn't seem like an O(N) thing to me, but granted.

Honestly, couldn't tell you what the failure rates are for SOTA in the real world as I don't work in that industry and numbers about anything DRM-related are kept quiet. Single digit error rates would be fantastically useful to the kind of person that thinks that's a good technology though, and it's usually pretty tunable.

[perching_aix]

> Persistent media watermarking through the analog hole is a solved problem and has been for years. It's standard practice on films.

Can you expand on that a bit? Wikipedia's coverage on this seems mostly historical and copy protection focused.

[AlotOfReading]

The basic idea is that you apply a very, very large amount of error correction to the tag and inject it into the media so that enough survives the severe geometric, color, and luminance distortions of a camcorder to recover the data out the other end. You then download the pirated cams and sue the theater.

There's a fair bit of public information out there on theoretical techniques (e.g. https://www.intechopen.com/chapters/71851), but I'm not deeply familiar with what's actually used in industry, for example by imatag.

[perching_aix]

Interesting, and the paper is surprisingly accessible to read as well, thanks.

One critique I can lodge against this is that to me it reads like the security model in this scheme trusts the venue to not tamper with the projection equipment. This may not map well to everyday camera recording situations, where the camera owner / operator may have a vested interest and capability in tampering with the camera itself.

[blibble]

a capable university student with their lam equipment could easily extract that key

I don't think the CIA will have any problems

[kylestanfield]

I hope you’re right