Hacker News new | ask | show | jobs
by bno1 454 days ago
What stops a scraper from detecting Anubis and just removing "Mozilla" from the user-agent string?
2 comments

That would allow you to specifically lock out that bot based on its user-agent string. That's the main problem with AI scrapers, many of them normally use user-agents that cannot be easily blocked, so other means have to be found to keep them off your grounds.
It breaks websites when you do that.