[jdrock]

True story: We crawled them a while back (before they expanded their engineering team) and because of our distributed system, "alarms" were going off. Rather then take time to tell their system we were not a DDOS attack, they put us in robots.txt. I imagine their small team had other stuff to work on.

[xtacy]

So, 80legs caused more traffic volume to reddit than Google crawler? That seems interesting; why would it be the case?

[jdrock]

Not necessarily more traffic, though that's possible. More likely it was the fact that it was coming from multiple IP addresses.