Hacker News new | ask | show | jobs
by BasiliusCarver 1420 days ago
One of the things I’ve done before among the other suggestions is to put a hidden link like /followmeifyouscraping.html in the landing page to get a bit of info about scraping volume and then you can use fail2ban filters to block if it’s visited if you want
1 comments

I would add that link to robots.txt as an exclusion.

That way well-behaved search engines won’t be affected, but naive scrapers get auto-banned.