[amelius]

Can't they pull the data from archive.org?

[KuiN]

Archive.org was knocked offline the other day due to some AI startup scraping it to death. It’s not a good thing.

[moneywoes]

Source, they don’t rate limit

[Kon-Peki]

https://news.ycombinator.com/item?id=36110527

[piperswe]

True - and their lack of rate limiting ended up letting someone overwhelm their servers, knocking them offline.

[edgyquant]

They put out a blog asking people not to scrape afterwards. A simple google will be much fast than asking for sources.

[SllX]

Archive.org is a non-profit without the capacity to serve that many requests. An excellent resource for people to use carefully, but not a treasure trove for bots to scrape down to the last bit.

[notpushkin]

Would be cool if they introduce some reasonably priced access for mass scrapers. Should make some nice income in addition to donations, and a valuable service to community.

[notacoward]

That would be worse.