Hacker News new | ask | show | jobs
by jlouis 3622 days ago
Another good trick is to melt the fuse if the request is merely slow. This can often back off before the timeout scenario occurs. I used this to great advantage when I crawled all duels of Quake Live years ago. If the site became slow, I would cease crawling for 15 minutes, and fuse handled that nicely.