[kuschku]

I agree. Part of what makes the web the web is that anyone has to be able to crawl its data, and build new technology on top.

As long as Cloudflare blocks that, it's hostile.

Maybe a way to register as spider with Cloudflare, and get a token that one passes in the header, with strict rate limits would be much better than the current solution of just showing captchas.