[kanzure]

> They hit 10 petabytes two years ago* - which is 10000 terabytes. Torrents don't really cut it at that scale.

I see no reason why torrents couldn't handle this. Data transport through bittorrent still works even with large numbers of bytes.

[pronoiac]

At these scales, bandwidth to disks isn't trivial - at a gigabyte a second, it would still take over 130 days to build the checksums for the torrent. I don't think Bittorrent fundamentally allows multithreaded torrent generation.

I think they bundle their data into WARCs of a few (dozen?) megabytes. Copying those around is much more straightforward, with much lower latency.

[zamalek]

I think he could have meant that some DHT could have been used, it needn't be specifically torrents. You could then volunteer your machine to redundantly store, say, 5GB of the archive.

If you are willing to spend time copying files around you should donate: your pay-per-hour probably far exceeds the time it would take you to dilly dally with billions of WARCs. Even if you flip burgers for a living. There are far better ways to contribute than by typing 'cp'.

[pronoiac]

You could use floppies to transfer terabytes of data, but that doesn't make it a good idea.