Y
Hacker News
new
|
ask
|
show
|
jobs
by
no_time
79 days ago
I would also like to know why is it excluded from Archive.org
https://web.archive.org/web/20260000000000*/https://www.true...
2 comments
bombcar
79 days ago
This can be done by Archive.org doing it for whatever reason (asked, on their own, etc) or it can be triggered by the current owner of the domain modifying robots.txt I believe.
link
T-A
79 days ago
Yes:
https://gist.github.com/danielhickman/38d6c7599d71c3e7dc48ca...
link
Thorrez
79 days ago
https://www.truecrypt.org/robots.txt
link