|
|
|
|
|
|
by bilekas
617 days ago
|
|
|
> does not respect robots.txt research shows. It would be nice then for the investigators to help people with the identifying markers for such crawlers. Apart from a mention of darkvisitors, which it seems is a paid service to "Block agents who try to ignore your robots.txt" I'm not sure how much that could be trusted given their business model also. |
|
|