|
|
|
|
|
|
by korkybuchek
297 days ago
|
|
|
> Google's bot was one of the few well behaved ones and would even slow scraping if it saw a spike in the response times. Google has invested decades of core research with an army of PhDs into its crawler, particularly around figuring out when to recrawl a page. For example (a bit dated, but you can follow the refs if you're interested): https://www.niss.org/sites/default/files/Tassone_interface6.... |
|
|