Hacker News new | ask | show | jobs
by fukd 1813 days ago
Back in 2018 i crawled 200Million urls of single site by using just freely avilable proxies on the internet to evade rate limits. i do not think it can be repeated today.

CDNs are blocking even my genuine requests.

1 comments

Sadly, it's easier than ever. Multiple botnets create open proxies which are easily discoverable, if you know where to look. Also, cloud providers give you more IPs you might ever need.
Cloud provider ip ranges are publicly known. They are the ones first get banned