[binarymax]

It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.