Hacker News new | ask | show | jobs
by robbya 2347 days ago
https://academictorrents.com/about.php#mirroring

Using RSS to allow mirrors to host different subjects is really clever, although some of the categories seem quite large (>5TB). It may be worth breaking up each category (sharding) to keep each to 100GB or less so a volunteer can pick a couple and not worry about running out of disk when a category grows.

Then it would be good to track how many seeds each category-shard has so volunteers can help where it's most needed.

1 comments

Some individual items are multiple TB, which would make 100GB shards a little difficult.