|
|
|
|
|
by jrwr
1633 days ago
|
|
They do respect robots.txt matter of fact, it will remove the website outright and most of its archives will be hidden until that website's robots.txt is offline. Their documentation about it is rather crap right now, but its in several of their FAQs about it |
|