[coldbrewed]

I work at a CDN that provides bot detection services. I agree that there's baseline necessity in terms of fraud detection, and if not necessity then definitely financial motivation to fingerprint. But these days, abusive scraping is far and way the the main driver for fingerprinting.

We don't fingerprint for ad purposes, and we destroy PII for humans as fast as we can because PII should be treated as radioactive. But we see customers that are constantly burned by abusive scrapers and the scrapers aren't slowing down.

The current approach to scraping is strip mining the Internet and is having the corresponding pollution effects that you'd expect. I'm fine with individuals doing whatever weird automation they want, more power to you, but it's this industrial scale crawling and extraction that's degrading the Internet from all angles.

[0xbadcafebee]

> we see customers that are constantly burned by abusive scrapers and the scrapers aren't slowing down

So, I have two dumb questions:

1) Can't the customer rate-limit connections? If the "abuse" in scraping is the number of requests... limit the number of requests?

2) There is probably a market now for federated authentication where the provider gives legal guarantees of anonymized fingerprint in exchange for either payment or selling data. This would provide an identifier for rate-limiting requests, and be pretty much the same as "Sign in with Google", but without identifying the user to the customer site. Are there too many problems with this that make it unfeasible, or would your company/customers be open to such a solution?

[fn-mote]

> Can't the customer rate-limit connections?

Bad guys use a botnet (compromised residential computers and routers) to defeat rate limiting and ip-based blocking.

[cute_boi]

Recently there was a article where LG smart tv were hacked and used for this entire thing.

[coldbrewed]

Exactly this. Residential proxies are _the_ tool that makes stealth browsers and massive scraping work, and it's overwhelmingly done without informed consent.

[mvdtnz]

> If the "abuse" in scraping is the number of requests... limit the number of requests?

Step one in doing this is identifying which requests are coming from the same agent. Guess what fingerprinting helps with.

[sunnybeetroot]

1) Scrapers often rotate ip per request, rate limiting doesn’t help

[codedokode]

Rate limiting requires them to buy more IPs (which are not free) and slows down scraping.

[sunnybeetroot]

Yes, that’s what the big scrapers are doing (frontier AI labs). They have infinite money for this.

[tormeh]

Do you think this could drive a shift towards more efficient web tech? I.e. more static websites, or websites served by faster software?