Hacker News new | ask | show | jobs
by xena 413 days ago
Do you respect robots.txt? How can I block your crawlers?
1 comments
skylerwiernik 413 days ago
Do you find this to be worse than googlebot somehow?
link