|
|
|
|
|
|
by s0me0ne
4045 days ago
|
|
|
The Internet Archive will still archive your site even if you use their recommended robots.txt suggestions to have them not archive your site. I had several domains where I setup the robots.txt immediately as one of the first things on my sites. While I owned the sites and ran them with the robots.txt Archive.org would not show any archive of them. However once the sites went down/changed domains, etc, the archive of them is now on archive.org. So all they do is block you from seeing what is on the site as of now, not forever. So you'd have to own the domain infinitely to have it not archived by them. |
|
|