[jgalt212]

Does Curl performance really matter? i.e. if it's too performant, doesn't that increase the odds your spider is blocked? Of course, if you're sharding horizontally across targets, then any performance increase is appreciated.

[j16sdiz]

libcurl is the backend for many (RESTful) API library.

Improving upload throughput to S3 bucket would be great, right?

[zamadatix]

What if you're not using curl as a spider? Even if you are I'd recommend some other spider design which doesn't rely on the performance of curl to set the crawling rate.