Hacker News new | ask | show | jobs
by Karhan 5478 days ago
Is there a good reason that reddit has banned 80legs crawling?
1 comments
jdrock 5478 days ago
True story: We crawled them a while back (before they expanded their engineering team) and because of our distributed system, "alarms" were going off. Rather then take time to tell their system we were not a DDOS attack, they put us in robots.txt. I imagine their small team had other stuff to work on.
link
xtacy 5478 days ago
So, 80legs caused more traffic volume to reddit than Google crawler? That seems interesting; why would it be the case?
link
jdrock 5478 days ago
Not necessarily more traffic, though that's possible. More likely it was the fact that it was coming from multiple IP addresses.
link