Hacker News new | ask | show | jobs
by rrtwo 3907 days ago
How does BuiltWith decides which sites to crawl? assuming it is too complex to crawl everything.

And how is the data/queries being sold without compromising it...

1 comments

I believe that they started via user submission of URLs to crawl. I used to submit tons of sites that I was curious about and eventually got their Chrome browser extension so I could find out the tech behind a site by clicking that little button.

Not sure if this is how they scaled out to the volume of sites they now cover.