Hacker News new | ask | show | jobs
by nkozyra 853 days ago
Although in general I like the idea of a queue for a scraper to access separately, another option - assuming you have the storage and bandwidth - is to capture and store every requested page, which lets you replay the extraction step later.