|
|
|
|
|
|
by pabs3
84 days ago
|
|
|
ArchiveTeam definitely do not intend to kill websites with too fast crawling, but definitely have done that unintentionally and always will stop/slow the crawling when it happens. Even the distributed crawling system has monitoring and controls to ensure it doesn't kill sites. |
|
|