|
|
|
|
|
|
by dredge
4832 days ago
|
|
|
No, it doesn't. Your linked example contains: User-agent: *
Disallow: /
While the (new) digg.com/robots.txt contains: User-agent: *
Disallow:
Those are very different. The former essentially disallows bots from crawling the entire site, while the latter disallows nothing - effectively allowing everything. The syntax is unusual, granted, for historical reasons. |
|
|