|
|
|
|
|
|
by tecleandor
461 days ago
|
|
|
You're getting it the wrong way. It's: Any crawler that's not well engineered, that doesn't follow robots.txt, that fakes its User Agent, that doesn't allow you to contact hem, that fetches content an indiscriminate number of times, repeatedly, all day long... can do this to your infrastructure unless you're a giant. What these crawlers are doing is akin to DDoS attacks. |
|
|