Hacker News new | ask | show | jobs
by lifeguard 4931 days ago
That is the way to block spiders that obey the standards and enough do not that robots.txt is not a solution.