Y
Hacker News
new
|
ask
|
show
|
jobs
by
zamadatix
874 days ago
What if you're not using curl as a spider? Even if you are I'd recommend some other spider design which doesn't rely on the performance of curl to set the crawling rate.