[nedwin]

I believe that they started via user submission of URLs to crawl. I used to submit tons of sites that I was curious about and eventually got their Chrome browser extension so I could find out the tech behind a site by clicking that little button.

Not sure if this is how they scaled out to the volume of sites they now cover.