[0xbadcafebee]

> we see customers that are constantly burned by abusive scrapers and the scrapers aren't slowing down

So, I have two dumb questions:

1) Can't the customer rate-limit connections? If the "abuse" in scraping is the number of requests... limit the number of requests?

2) There is probably a market now for federated authentication where the provider gives legal guarantees of anonymized fingerprint in exchange for either payment or selling data. This would provide an identifier for rate-limiting requests, and be pretty much the same as "Sign in with Google", but without identifying the user to the customer site. Are there too many problems with this that make it unfeasible, or would your company/customers be open to such a solution?

[fn-mote]

> Can't the customer rate-limit connections?

Bad guys use a botnet (compromised residential computers and routers) to defeat rate limiting and ip-based blocking.

[cute_boi]

Recently there was a article where LG smart tv were hacked and used for this entire thing.

[coldbrewed]

Exactly this. Residential proxies are _the_ tool that makes stealth browsers and massive scraping work, and it's overwhelmingly done without informed consent.

[mvdtnz]

> If the "abuse" in scraping is the number of requests... limit the number of requests?

Step one in doing this is identifying which requests are coming from the same agent. Guess what fingerprinting helps with.

[sunnybeetroot]

1) Scrapers often rotate ip per request, rate limiting doesn’t help

[codedokode]

Rate limiting requires them to buy more IPs (which are not free) and slows down scraping.

[sunnybeetroot]

Yes, that’s what the big scrapers are doing (frontier AI labs). They have infinite money for this.