[colinsane]

just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.

[binarymax]

It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.

[dylan604]

If you scrape 6TB from across the web or grab it from one place, it's still 6TB