[gerenuk]

What approach would you be going for initially for deduplicating same urls?

[verdverm]

You might look at a real data processing system, like something from the Apache projects