|
|
|
|
|
|
by vertis
804 days ago
|
|
|
Except now it says # silly bing
#User-agent: Amazonbot
#Disallow: /
# buzz off
#User-agent: GPTBot
#Disallow: /
# Don't Allow everyone
User-agent: *
Disallow: /archive
# slow down, dudes
#Crawl-delay: 60
Which means he's changing it. The default for all other bots is to allow crawling. |
|
|