[thwd]

Arguably, UploadToS3 should not be a method of *Payload.

Suggestion: Make an S3-uploader package with _internal_ connection pooling, upload queueing and concurrency handling.

[zimbatm]

This. Each S3-uploader can then hold a http.Client instance that keeps the connection open to S3 between uploads.

[placeybordeaux]

I haven't used this library, but this appears to be what you want. https://github.com/rlmcpherson/s3gof3r

Nice features include:

> Retry Everything: All http requests and every part is retried on both uploads and downloads.

> Configurable conncurrency

> Uses an io.Writer (you could actually start posting to S3 before all of the data gets in on your side.)

etc.