[ccgreg]

> and the data that I’ve experimented with from 2014 seemed high quality

That's because it's from the blekko search engine.

[n1xis10t]

Sounds like blekko had a larger impact on the early urls than I thought. Out of curiosity, do you remember how large blekko’s index was at it’s peak?

[ccgreg]

The largest index we had was 4 billion, which is tiny. Our crawl frontier was much larger.