[nathanb]

That would not solve the problem they're trying to solve.

Let's say I post something that I shouldn't have posted -- insider stock information, nude photos, whatever. Perhaps something illegal for me to post. I need to make it go away.

I need to be able to create a robots.txt today which affects stuff I posted yesterday.

This is why archive.org respects the current robots.txt for access to past content.