|
|
|
|
|
|
by jakopo87
1248 days ago
|
|
|
Years ago i added a url to not be crawled nor indexed to a web app I worked on to block bad crawlers, so I disallowed crawling through robots.txt and added a hidden link in the footer with the "noindex" attribute. Google, Bing, Archive.org, and people normally browsing were just fine while bad crawlers got a blank page. |
|
|