|
|
|
|
|
|
by groovybits
2529 days ago
|
|
|
I think the concern is more accurately: you must go out of your way to honor robots.txt. I think both robots.txt and security.txt are great ideas. However, they will always only be useful to those who follow the wishes of the website (which hopefully outweight those who do not). |
|
|