|
|
|
|
|
|
by syxun
2529 days ago
|
|
|
>If you write a crawler, you probably don't want it to waste time indexing a list of articles in every possible sort order, trying all "reply" buttons, things like that. On the other hand, many websites (like wikipedia here) hide interesting pages behind a Disallow. |
|
|