[wordpad]

Why can't crawling be crowd sourced? It would solve ip rotation and spread the load

[6510]

https://yacy.net

[catlikesshrimp]

Too bad it doesn't support android. It is much more energy efficient than anything else I can spare (for 100% uptime contribution)

[Poomba]

That’s how residential proxies work, in a perverse way

[chiefsearchaco]

Common crawl sort of serves this function. I use it. It's a really good foundation.