Hacker News new | ask | show | jobs
by deadalus 1740 days ago
archive.org obeys the robots.txt exclusion but archive.today doesn't. This means that many websites(like 4chan) cannot be archived with archive.org.
1 comments

That's true, so does webcitation.org, which doesn't obey robots.txt.

4chan has archives dedicated for that site anyway (Warosu, etc.).