Hacker News new | ask | show | jobs
by deferredposts 3363 days ago
So what is the policy of The Internet Archive on this level of scraping? Do they have a rate limit in place?
1 comments

Yes, they start sending 429 (Too Many Requests) responses if you don't use appropriate delays. They also provide a public API [0] which I believe is intended for automated requests of this type (as opposed to crawling the Wayback Machine website directly).

[0] - https://archive.org/help/wayback_api.php