Hacker News new | ask | show | jobs
by zamadatix 874 days ago
What if you're not using curl as a spider? Even if you are I'd recommend some other spider design which doesn't rely on the performance of curl to set the crawling rate.