|
|
|
|
|
|
by greglindahl
2530 days ago
|
|
|
That's not true in the actual web, however. The best example is a large number of unimportant sites that send 429 errors for /robots.txt if they think it's a scraper. A 4xx result for robots.txt is considered to mean no robots.txt for most crawlers. So the website is getting the reverse of what it thought it was getting. |
|
|