|
|
|
|
|
|
by nuc1e0n
1032 days ago
|
|
|
robots.txt is a standard and has been for many years. It seems that https://www.robotstxt.org is the main repository for its definition. Any robots.txt I might suggest wouldn't necessarily be appropriate for anyone else. It's a policy that site stakeholders need to decide for themselves I'd say. |
|
|