Hacker News new | ask | show | jobs
by chillaxtian 2764 days ago
S3 Multi-Part Upload API can be used to chunk an object into smaller parts, which can succeed or fail independently.

https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview....

1 comments

Yes that is very helpful. Our s3 storage backend for tusd uses it, and our https://uppy.io file uploader does too, usable directly from the browser (so you can choose to not use tus at all with it). S3 resumable uploads do come with a few limitations that make some people still choose tus tho:

* chunks need to be >5MB which can be problematic on flaky/poor connections (rural areas, tunnels, clubs/basements, people on the move switching connections all the time)

* your s3 bucket needs to allow write by the world, or you need to deploy signature authentication

* there’s an s3 vendor lock-in some might worry about

* not an open protocol, no chance of advancing it with the community

That said, that still leaves a large audience for direct s3 resumable uploads and I’m thankful aws offers it!

As far as vendor lock in, it seems like there are a large number of other vendors supporting the S3 API, so this doesn't seem like a huge concern.
That’s a fair point. And I guess with e.g. Minio you could selfhost too.

S3 is great and in fact, at Transloadit we deploy a content ingestion network (reverse cdn) of many regional tusd servers, close to our customers’ end users, but they all ultimately save to S3 using multipart. We’re happy S3 customers.

So why the extra layer. Because this let’s us offer resumability below 5MB, lower regional latencies, roll our own auth, and switch to a different cloud provider without introducing breaking changes at the customer facing side (assuming the new cloud bucket provider does not offer an S3 compatible interface, or even just a slightly incompatible one)

Ultimately you’re still locked-in with AWS protocol-wise, and there’s no community platform for advancing it, so addressing any of these issues is going to be hard.

My comment wasn't intended to a dig at tus. Like the work you all are doing!
<3