Y
Hacker News
new
|
ask
|
show
|
jobs
by
walterbell
4265 days ago
Is there a generic list of defense techniques against scrapers, including but not limited to Google? E.g. move some content behind a login wall?
1 comments
landtco
4265 days ago
In theory, you can disallow crawlers and bots via robots.txt. It's not a hard defense per se because crawlers can ignore it.
link