[tptacek]

The unsatisfying answer to this is probably that it just doesn't matter. It's not as if evidence chain of custody is assured cryptographically; it's assured by rules and regulations and an adversarial system. If you tried to submit as evidence a forged document vouchsafed with a colliding MD5 hash, you'd be putting your own freedom at risk, because the forgery will be straightforwardly detectable (the real document won't have hash colliding artifacts in it).

None of this is to say that the legal profession shouldn't move to SHA2; it should.

[colmmacc]

You say to use SHA2, but TFA says to use SHA3 or Blake. I think your recommendation is the better one, but I feel like teasing out why because it's interesting.

Firstly ... the NIST recommendation TFA links doesn't just recommend SHA-3, it actually says "Federal agencies should use SHA-2 or SHA-3 as an alternative to SHA-1." SHA-2 and SHA-3 are both valid and recommended hash functions by NIST. And while 3 is higher than 2, and SHA-3 is newer, in this case it doesn't mean "better". Being based on Keccak and a sponge construction, SHA-3 provides "diversity" more than "improvement".

Secondly ... SHA2 is widely implemented in existing hardware, and it's just currently more efficient (and likely to remain so). So why waste power, especially on something you'll be doing in bulk.

O.k., so that's why SHA-2 and not SHA-3. But Blake is worth avoiding IMO ... because the FISMA says that for Federal work, you have to use one of what NIST recommends in FIPS. Obviously the legal profession needs to be able to practice in Federal courts (Article III and administrative) ... so if you're going to pick a new standard, pick one of those (but not SHA-3!).

Lastly, and this is really an aside ... it's not uncommon for some folks to think SHA3 and SHA384 are the same thing, but they are not. SHA384 is just a variant of SHA-2 with a 384-bit digest length and correspondingly improved security margin. Other Federal standards, like CNSA, separately recommend SHA384 as a good minimum ... so it can be confusing and I think it's understandable why some people think this is what SHA3 is just short for.

[wolf550e]

BLAKE3 is faster than hardware accelerated SHA-2 because the tree mode used in BLAKE3 allows hashing parts of a single message in parallel (with SHA-2, parts of a single message have to be hashed one after another, and parallelism is only used in workloads where you process multiple messages at the same time).

https://github.com/minio/sha256-simd

https://github.com/BLAKE3-team/BLAKE3

[adrian_b]

SHA-3 (also BLAKE2 or BLAKE3) is definitely more secure for very large documents than SHA-2.

The security (i.e. the difficulty in finding collisions) decreases with the length of the document for SHA-2 and it stays constant for SHA-3.

Nevertheless, it is unlikely that typical legal documents are big enough for this to matter, except when the hashes would be e.g. for entire seized HDDs or SSDs, so SHA-2 is an acceptable choice for replacing MD5.

The only reason why SHA-3 has not become widespread is that it, like also AES, requires hardware support for good performance, but for some reason Intel has not added an SHA-3 instruction to the x86 ISA. Arrow Lake S, expected to be launched at the end of this year, will add support for SHA-512 and for the Chinese standard hashes, but there is still no intention to add SHA-3 (like Arm already did).

Both AES and SHA-3 are not recommendable on CPUs without dedicated instructions, due to low performance, but with hardware support they become faster than any alternatives. The difference between them is that now only the cheaper microcontrollers lack AES support, while SHA-3 support is still seldom encountered.

[oconnor663]

> The security (i.e. the difficulty in finding collisions) decreases with the length of the document for SHA-2

Could you spell this part out for me?

[adrian_b]

Twenty years ago this paper was considered surprising and, together with a handful of other attacks and with the concrete attacks against MD5 and SHA-1 succeeded by some Chinese researchers prompted the organization of the SHA-3 competition.

John Kelsey and Bruce Schneier: "Second Preimages on n-bit Hash Functions for Much Less than 2^n Work"

https://eprint.iacr.org/2004/304

The abstract at this link provides the essential results.

"We provide a second preimage attack on all n-bit iterated hash functions with Damgaard-Merkle strengthening and n-bit intermediate states, allowing a second preimage to be found for a 2^k-message-block message with about k * 2^(n/2+1) + 2^(n-k+1) work. Using SHA-1 as an example, our attack can find a second preimage for a 2^60 byte message in 2^106 work, rather than the previously expected 2^160 work."

Besides this result, there is also the previous result obtained by Joux for multi-collisions, which also become easier for longer input data (Antoine Joux: "Multicollisions in Iterated Hash Functions. Application to Cascaded Constructions").

[oconnor663]

Thanks! I've added a note about this here: https://github.com/oconnor663/bao/issues/41#issuecomment-119.... Does that sound like an accurate summary to you?

[adrian_b]

It is OK, but the problems demonstrated by these attacks and others similar to them are specific to the linear chaining of a great number of blocks, where the same hashing function is applied to all blocks.

Incorporating somehow a block counter in the block hashing function or using a sponge structure is equivalent to using a different hashing function for each block, which stops the attacks.

Tree hashing where the final branches i.e. the chunks are short, so there are only a small number of blocks hashed in cascade with the same function, behaves differently from linear block chaining, because the hashing function must include additional inputs anyway, so the blocks in a chunk are hashed differently from the blocks that merge tree branches or the root block.

Whether any of the attacks that exist for simple Merkle-Damgaard cascading like SHA-2 are applicable to a tree hash depends on how the blocks are encoded and on the structure of the tree. In any case, the probabilities of success will be different in the case of a tree hash.

I have never analyzed whether the BLAKE2 or BLAKE3 tree hashes could be secure enough without block counters. In any case, the block counters, which are absolutely necessary for linear block chaining, also increase the strength of tree hashing against any attacks and their added overhead is minimal, so there would be no reason to remove them.

A hash computed with a sponge structure, like SHA-3 or KangarooTwelve, is equivalent with a CBC-MAC where the key is updated at each block, instead of being constant. Because each block is hashed with a different key, there is no need for an additional counter input to differentiate the hashing functions. A sponge structure is much simpler than the structure used by BLAKE, but it needs a wider invertible mixing transformation, e.g. for a similar strength BLAKE2b uses an 1024-bit mixing transformation, while SHA-3 uses an 1600-bit mixing transformation. (BLAKE3 trades off strength for speed, so it is not comparable with SHA-3, but only with KangarooTwelve.)

[akerl_]

It feels like you're just hunting for complexity a bit here. Any of SHA2, SHA3, or Blake are viable options.

I'd only let NIST and FIPS drive my crypto choices if I was being actively forced to do so by a government use case, and even so I'd be praying for the day that FIPS joins us in the modern era and stops tethering us to acronyms they could enumerate a decade ago.

[tptacek]

He's hunting for complexity because it's interesting to talk about!

[DannyBee]

Lawyer here. This is right. Unlike some of the typical software use cases, md5 is usually not the only thing stopping you from falsifying evidence.

The records being produced almost always exist and the hash is being used to differentiate which is which and whether you got are correct copies.

If evil litigant produces fake documents instead of real ones, and you can't get access to the system they came from, no hash will save you - they can produce any hash they want and you can't verify against an original.

If you can verify against the original, forensics isn't going to look at a hash and say "hash matches my job is done".

Most of the time what matters is whether the evidence exists or not.

The tobacco companies lied and claimed they never had any evidence in the first place.

assume instead they wanted to falsify the data to say that as far as they could tell smoking did not cause cancer.

this is not a hashing problem. This is a problem of making up fake studies that look real.

[woodruffw]

This is more or less what I learned when I worked on forensics software, the kind that was supposed to maintain this kind of chain of custody/integrity. Like most things that touch the legal system, the presumption is that dishonesty or unsoundness in the chain of custody is fundamentally a legal problem with legal recourses, not something that can be solved with math.

[graypegg]

I quite like the licensing trick Nintendo used on the gameboy as an example of this. [0]

Essentially, the gameboy expected a bitmap of the Nintendo logo to be present on the cartridge rom, and was shown on screen at boot. It had to match a version stored on the gameboy itself or else the game wouldn’t start.

The thinking (that I’m not sure was ever tested) was that someone producing a game that tried to trick consumers into thinking it was an official Nintendo product, would be liable for damages in a trademark lawsuit. Since the game would never start without an official Nintendo logo, the hope was to make the legal system enforce Nintendo’s licensing scheme.

[0] https://catskull.net/gameboy-boot-screen-logo.html

[bentley]

The thinking was tested (in U.S. jurisdiction) in Sega v. Accolade. https://en.wikipedia.org/wiki/Sega_v._Accolade

The court sensibly ruled that using technical means to force competitors to display your trademark against their will doesn’t mean you can then claim they’re infringing that trademark.

[pcwalton]

Apple tried this too with Dont Steal Mac OS X.kext, which uses a haiku with a copyright message as the key to decrypt certain executables like the Finder. I don't think it had any real-world impact.

[kevin_thibedeau]

Pretty rich considering their history with sosumi.

[radarsat1]

Also reminds me of MODULE_LICENSE and EXPORT_SYMBOL_GPL in the Linux kernel.

https://www.kernel.org/doc/html/latest/process/license-rules...

[DannyBee]

Right. If you don't have access to the original system (whether it's the thing that stored the emails or a computer with evidence on it), just an image, the hash doesn't matter anyway. They can just put fake data in the evidence vault system to begin with.

If you do have access to the original system, md5 is usually not the thing in the way of falsifying evidence.

most of the time they will claim, the evidence does not exist at all, rather than try to falsify it.

falsification requires making lots of things that makes sense historically, and humans to swear to them.

you also often have to falsify more than one system in a consistent way.

you have to do all of these things in a way that the forensic specialist is not going to think that everything looks really weird

[NoboruWataya]

Exactly. Both sides will have copies of all pertinent evidence, and if their copies conflict in any material way, it will be noticed and investigated and the party that manipulated their copy is going to have a very bad time.

The examples given in the article rely on an attacker manipulating an image prior to it being hashed, or compromising their opponent's database. If you can (and are willing to) do these things, no hashing algorithm will help. They also assume that the hash is literally the only thing anyone will rely on in identifying a document, which is not how it will work in practice. (No witness has ever said "yes there was a letter, I don't remember what it looked like or what it said but I remember its MD5 hash was [...]".)

[planede]

> the real document won't have hash colliding artifacts in it

I don't think that hash colliding artifacts would necessarily be obvious. They could be in part of a file that ends up being ignored by parsers of the file format. Or it could be some low level noise in pixel values in scanned documents.

[franga2000]

Both of those are things computer forensics experts are used to looking for and I'm certain even a beginner could find them.

[planede]

Are court documents routinely analyzed by computer forensic experts, or only when there is suspicion of tampering?

[singleshot_]

It certainly does not matter. If you wanted to present a fake document as evidence, would sha-whatever help? Nope. If you wanted to withhold key evidence and pretend it didn’t exist, would it help? Nope.

And do you think we trust md5, or do you think we have another tool that can compare documents? We do have such tools.

What prevents chicanery in discovery and disclosure is how annoying it was to go to law school and pass the bar compared to how fun it would be to work at Wendy’s for the rest of our careers.

Should we use a better hash function: probably. Is this a problem of not enough technology literacy in the profession? Well, not really; there are plenty of tech literate lawyers (hi!) we just have questionable tools just like you do in IT and we make do.

[brohee]

What if both documents have collision artifacts? That's the most realistic attack in e.g. a contract dispute.

[paulddraper]

That's right.

Same reason why blockchain is (almost) never the answer: that's not how human systems work.

Trust is centralized. Always had been.

[lnxg33k1]

I would say the system understands MD5 being broken, but more of usability issue the system is leaving a backdoor to keep itself safe, comes to mind how the system got threatened about Epstein and went over and under in order to keep him free, society should be worried about legal system having backdoors