|
|
|
|
|
|
by paulie_a
2546 days ago
|
|
|
Not intentionally ignore it by going out of their way to override it, just not be required to implement a feature to their crawler. Apparently parsing those sounds tricky with edge cases. Ignoring that file is absolutely on the table. People of course can adhere to but it's not required and in my opinion shouldn't even be paid attention to. In my younger years the only time I ever dealt with robots.txt was to find stuff I wasn't supposed to crawl. |
|
|