|
|
|
|
|
|
by TrackerFF
37 days ago
|
|
|
Is it just me, or is it extra unethical and self-serving when crawlers from say Amazon(Bot) decides to incessantly crawl AWS hosted websites? Same goes for Google and Microsoft crawlers crawling GC and Azure. By that, I mean the types of crawls that can hog up significant usage. |
|
|