Hacker News new | ask | show | jobs
by buro9 619 days ago
Also the Facebook hit scraper.

Which does not respect robots.txt and definitely is just scraping.

AS blocks are the only really effective tool now, there are many scrapers that do not even respect user agent

1 comments

Facebook and someone who’s using a Firefox UA are the big hitters for me today, each at a sustained 6 req per second for the last 24 hours on one site.

Today is actually pretty good, there’s some real looking UA traffic in the top 10.