Hacker News new | ask | show | jobs
Songkick's robots.txt has a few special rules. (songkick.com)
1 points by nintax 5165 days ago
1 comments
dibarra 5164 days ago
It's interesting they're disallowing 008. I work at a popular webhost, and we've had problems with this crawler too (It's distributed, and can crawl things a little too fast).
link
nintax 5164 days ago
Did you notice the bottom 3 rules? Maybe I should resubmit and change the title...
link