The worst part is when a site is stored in the wayback, then the domain expires and the new owner (or squatter) has a robots.txt that blocks everything, then all the old content becomes inaccessible.
They should store a history of WHOIS data for the site's domain and make separate archives for when the owner changes, I think. Also, why did anyone think that applying robots.txt retroactively is a good idea? :/