Hacker News new | ask | show | jobs
by TeMPOraL 3904 days ago
There's a really bad problem with archive.org - that they retroactively apply robots.txt files. Simple:

    User-Agent: *
    Disallow: /
And poof, here goes history. Guess how many companies care?
1 comments

> Guess how many companies care?

Many. But, more in the way of actively and enthusiastically supporting the ability to wipe the archive this way than, you know, being concerned that it is a problem.