[Jefenry]

Annoyingly, Internet Archive doesn't respect robots.txt. I specifically excluded ia_archiver from my site which worked for a number of years until they decided to ignore it because robots.txt "do not necessarily serve our archival purposes." They do remove your site if you email them though.

https://www.digitaltrends.com/computing/internet-archive-rob...

[seabass-labrax]

Personally, I'm with the Internet Archive on this one. If they were to respect robots.txt, it wouldn't be long before a whole host of websites exclude the Internet Archive for dubious reasons such as lost advertising revenue, copyright concerns, exclusivity deals etc. I am curious to know if you've found the Internet Archive's activity to be exceptionally taxing on your servers, or whether you have another reason to wish to exclude them?

[Jefenry]

Mainly I just feel I should be in control of the sites I make. They're personal in nature that I don't mind sharing with the world, but if I want to change something or make them disappear completely it irks me that there's a website out there violating my express wishes.

[seabass-labrax]

How interesting! That's a completely different way of looking at the Web; I don't think I've thought about it like that before. I view the Web as a kind of library where you can add books as well as borrow them.

When I read something online, I sort of feel that it becomes a part of me as it informs me and shapes my perspectives. I like to think that I could re-read it if I ever forgot the details; as a result, I've downloaded quite a few websites. Common facts don't really apply here for me, as they'll be accessible as long as encyclopaedias exist, but personal anecdotes and niche catalogues are worth their metaphorical weight in gold.

Additionally, I download things that I one day hope to read, but think that the website might disappear before that time comes (due to the author not renewing a domain name etc.)