Hacker News new | ask | show | jobs
by ecesena 4405 days ago
At Theneeds we use a "sliding windows" approach to deal with polling. Say you run the scraper every hour. Each feed F_i is scraped once every n_i polls. If the feed returns more than his average news, then n_i gets decreased, while if the feed returns no news, it gets increased.

Perhaps with a similar trick you can run your scraper more frequently on some feeds, still keeping the cost under control.