Y
Hacker News
new
|
ask
|
show
|
jobs
by
deadalus
1740 days ago
archive.org obeys the robots.txt exclusion but archive.today doesn't. This means that many websites(like 4chan) cannot be archived with archive.org.
1 comments
oenetan
1740 days ago
That's true, so does webcitation.org, which doesn't obey robots.txt.
4chan has archives dedicated for that site anyway (Warosu, etc.).
link
4chan has archives dedicated for that site anyway (Warosu, etc.).