|
|
|
|
|
by BasiliusCarver
1420 days ago
|
|
One of the things I’ve done before among the other suggestions is to put a hidden link like /followmeifyouscraping.html in the landing page to get a bit of info about scraping volume and then you can use fail2ban filters to block if it’s visited if you want |
|
That way well-behaved search engines won’t be affected, but naive scrapers get auto-banned.