|
|
|
|
|
|
by techdevangelist
2210 days ago
|
|
|
That’s close to my theory as well, Googlebot is almost universally allowed, building the index happened at the right time to become to big to block. Trying to deep crawl to Google’s level may be impossible now, and maybe one way to encourage innovation in search/discovery is to force the raw crawl results for others to parse & rank. |
|
|