[jimsimmons]

What is the best way to deduplicate a corpus of documents

[marginalia_nu]

If you mean in the sense of dealing with documents that are very similar but not binary identical, a locality sensitive hash would do the job.

[OnlyMortal]

A Rabin finger printing algorithm and a hash of the data it generates.

Reference count the hashes.