|
|
|
|
|
by microgpt
1 hour ago
|
|
Pretty easy to display one thing to verified browsers (just latest few user-agents from the 10ish different mainstream browsers on the 3 main OSes) and another to anything else. Yes AI scrapers can easily spoof user-agent, but they fall out of date as the browser updates. Bit harder to catch them in tarpits and then serve nonsense to whoever ever triggered the tarpit. |
|
It’s a hell of a lot easier for a company to ensure that its scrapers all report the latest user agent string than it is to get everyone and their mother to update their browsers in a timely fashion.