|
|
|
|
|
|
by snowcode
1450 days ago
|
|
|
more thoughts; Invert allow/disallow Robots.txt should be inverted, with everything disallowed by default, and only allowed paths to be configured. Requiring site owners to explicitly list dissalowed secret folders is ludicrous. How this became de-facto is hard to fathom. If a site owner wants to allow everything then he/she should have to explicitly add a global allow. |
|
|