Hacker News new | ask | show | jobs
by sunnybeetroot 20 days ago
1) Scrapers often rotate ip per request, rate limiting doesn’t help
1 comments
codedokode 19 days ago
Rate limiting requires them to buy more IPs (which are not free) and slows down scraping.
link
sunnybeetroot 19 days ago
Yes, that’s what the big scrapers are doing (frontier AI labs). They have infinite money for this.
link