|
|
|
|
|
|
by nemothekid
4117 days ago
|
|
|
Google shouldn't really have to do this. Replace GoogleBot with BingBot or GCE with AWS and you still have the same problem. A website operator should be working to make sure search crawlers don't consume too many resources given that the bots follow rules. Otherwise you'd have a team at every cloud provider trying to figure out how to manage bots. |
|
|