[verdverm]

This is easier than deduplicating the many different URLs which have the same content. A harder problem awaits you!

ML & basic stats

[gerenuk]

What approach would you be going for initially for deduplicating same urls?

[verdverm]

You might look at a real data processing system, like something from the Apache projects