[ericst]

Have you ever heard of https://commoncrawl.org/? Maybe it helps you to bootstrap.

It's cool to see some new things happening in the search engine world.

[brentadamson]

I've considered that but didn't really know anything about the organization and how long they'll be around. We're gonna give Colly a try: https://github.com/gocolly/colly.