Hacker News new | ask | show | jobs
by nreece 2339 days ago
Cool site! Curious to know what do you use for scraping, without getting blocked?
1 comments
sphix0r 2339 days ago
Thanks a lot! We use scrapy[1]. Our crawler's behavior is very polite so most of them won't block us. Easyjet for example is very strict against crawlers and will block most of them(and us).

[1] https://scrapy.org/

link
krn 2339 days ago
> Easyjet for example is very strict against crawlers and will block most of them(and us).

It seems that easyJet has outsourced its anti-scraping protection to the company called "Distil Networks"[1].

I wonder how could they block screen scrapers using Chrome on Android with mobile IP addresses.

[1] https://www.youtube.com/watch?v=43wh3beHSBo

link