Hacker News new | ask | show | jobs
by yutah 4313 days ago
Which large website did you get banned from for scraping? I did some scrapping and never got banned... perhaps my scraping's rate was not too fast.
1 comments

If you try to mass scrape almost any major site (millions of pages of content) they'll block you.

For example, if you went one by one through Stack Overflow and sucked out every question and answer, your scraper bot would get banned (unless you're doing one request per minute, in which case you'll never finish).

Or if you tried to scrape Twitter.