|
|
|
|
|
|
by r721
3353 days ago
|
|
|
>may cause the Internet Archive to be banned from crawling some websites. It looks like Facebook banned ia_archiver (recently? I recall it worked a few weeks ago): >User-agent: ia_archiver >Disallow: / https://www.facebook.com/robots.txt |
|
|