|
|
|
|
|
|
by kazinator
455 days ago
|
|
|
> The old style bots were rarely a problem. They respected robot exclusions and "nofollow" warnings. What year are they reminiscing about here, 1999? Nothing has respected robots.txt in over twenty years. Nofollow isn't an anti-bot measure; it's supposed to inform search engines that you don't vouch for the linked content (don't wish to boost its rank).
Nofollow doesn't mean "you must not follow this if you're a crawler". |
|
|