[JDye]

Web scrapers maybe aren't "bad actors", but many sites dont want them. They'll use tons of TCP proxies which route them through a rotating pool of end user devices (mobiles, routers, etc...). Its not really possible to block these IPs as you'd also be blocking legitimate customers so other ways to detect and block are required.

[dlenski]

Can't/won't these scrapers just switch to using VPNs or sshuttle or basically anything else that doesn't leak timing info about termination of TCP vs HTTP?

[JDye]

Not really. You can have 100,000 IPs from proxies or use VPNs and have only 5 egress IPs.

Anybody who wants to stop the scraper could get browser fingerprints, cross reference similar ones with those IPs and quite safely ban them as its highly likely theyre not a legitimate customer.

Its a lot harder to do it for the 100k IPs because those IPs will also have legitimate customer traffic on them and its a lot more likely the browser fingerprint could just be legitimate.

The risk of false postives (blocking real people) is usually higher than just allowing the scrapers and the incetives of a lot of sites arent aligned with stopping scrapers anyway. Think eccommerce, do they _really_ care if the product is being sold to scalpers or real customers? If anything, that behaviour can raise perception of their brand, increase demand, increase prices.

This tool should have less false positives than most, so maybe it will see more adoption than others (TCP fingerprinting for example) but I dont think this is going to affect anyone doing scraping seriously/at scale.

[dlenski]

> Not really. You can have 100,000 IPs from proxies or use VPNs and have only 5 egress IPs.

Why…?

If I can run a proxy exit node on 100k residential IPs, why can't I run a VPN server on 100k residential IPs?

There is no additional technical complexity or resource consumption from the VPN server compared to the proxy server.

[JDye]

I don't mean that you can't do it, just that there is no company offering it so right now those are the only two options.

It's something we're experimenting with currently. the other commenter is right about apple products, but on android, desktop, etc... it's pretty easy.

[benmmurphy]

for phones its a bit difficult because i don't think you can egress out ip traffic without root or jailbreak on iphone and iOS. but i guess on desktop this should be possible