Y
Hacker News
new
|
ask
|
show
|
jobs
by
Nemo_bis
2086 days ago
Thanks, so maybe this page is outdated where it mentions your own crawler with user-agent? Or does the Internet Archive use it for these crawls?
https://www.cloudflare.com/always-online/