Hacker News new | ask | show | jobs
by jiggawatts 1420 days ago
I would add that link to robots.txt as an exclusion.

That way well-behaved search engines won’t be affected, but naive scrapers get auto-banned.