|
|
|
|
|
by franze
2229 days ago
|
|
here the problem comes in, as people write User-agent: googlebot
User-agent: bing #behaved-badly
User-agent: yandex
Disallow: /
if # removes everythin afer the #, then it would be User-agent: googlebot
User-agent: bing User-agent: yandex
Disallow: /
which results into the whole line " User-agent: bing User-agent: yandex" beeing thrown out as malformed, so only googlebot would be disallowed. |
|