Hacker News new | ask | show | jobs
by bearjaws 668 days ago
At this point honoring robots.txt will ensure you have a terrible searching experience...