Y
Hacker News
new
|
ask
|
show
|
jobs
by
jiggawatts
1420 days ago
I would add that link to robots.txt as an exclusion.
That way well-behaved search engines won’t be affected, but naive scrapers get auto-banned.