Hacker News new | ask | show | jobs
by Nemo_bis 2086 days ago
Thanks, so maybe this page is outdated where it mentions your own crawler with user-agent? Or does the Internet Archive use it for these crawls? https://www.cloudflare.com/always-online/