[pseudalopex]

It was correct. And archive.org used this self inflicted problem to justify disregarding robots.txt.

https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

[npunt]

Oh thank you for the update, that's great news! Agree with their position 100%.

[pseudalopex]

> Agree with their position 100%.

Including the part where they blamed anyone but themselves for hiding old snapshots when robots.txt changed?

[npunt]

no I didn't see that in there, I just agree that robots.txt makes sense for web crawlers but not archival purposes