Hacker News new | ask | show | jobs
by bjord 1308 days ago
this won't stop the overall trend, but it can help you get around cloudflare's effective scraping blocking (copying my comment from a previous thread):

If you're scraping with Python, try cloudscraper—among other things(!), it supports JS rendering (basically the bare-minimum check cloudflare does), without needing to run a full browser in the background. It's built on requests, so integration was pretty easy.

https://github.com/venomous/cloudscraper

1 comments

JS rendering is not enough. Cloudflare monitors UI interactions and browser footprints to assess whether it’s a human or a bot.
trust me, I'm aware—cloudscraper can also solve cloudflare challenges, including turnstile