|
|
|
|
|
by jfr
5609 days ago
|
|
Except for the schema and host parts (which are not part of robots.txt anyway) URLs are case sensitive (ref: RFC 3986, sections 6.2.2.2 and 6.2.3). The problem here is that Microsoft servers respond to /search, /Search and /SeaRCh without distinction. They are all distinct URLs. If it was the intended behavior (stupid, but understandable, coming from Microsoft), then robots.txt should contain all variants in capitalization for each path. A better solution would be to force a 301 redirect to a canonical path, and have this path in robots.txt. Google would work as expected. The original article is totally bogus. I can't imagine how it has over 90 votes. |
|