Hacker News new | ask | show | jobs
by rad_gruchalski 566 days ago
Yeah, but you cannot multipart single chunk into a larger complete file. You need all chunks one way or another. Multipart upload starts and ends from all chunks. GCS and Azure support this too. S3 does a maximum of 1k objects,

GCS 32 objects, and Azure blob storage, afair 5k objects. Both can do an operation similar to what you described for S3 with various alternatives of read at offset + length and rolling those up.

In all cases, you end up always rolling up into a new key that isn’t available for read until roll up is done. It’s kinda useless for heavy write scenario.

Compare that to your normal fs operation. Write at offset to an existing file with size smaller that offset will just truncate the file to the offset, and continue writing.
1 comments
ncruces 566 days ago
You can?

You create a multi-part with 3 chunks: the 1st part is a copy range of the prefix, the 2nd part the bit you want to change, and the 3rd a copy range of the suffix?

And yes, all of this is useless for heavy (and esp. concurrent) writes.

link
rad_gruchalski 566 days ago
We both said the same thing. You kinda can but cannot. Yes, you can replace some part of an existing object but you cannot resize it, not can you do anything parallel with that. So you kina can but cannot. And this trick will work in gcs and azure, here you have to move the new object to an old key yourself after the roll up. But why not while you’re already at it.
link
ncruces 566 days ago
You can do it “in place” as the target can be the same as the source. And you can definitely resize it, both truncate it and extend it. The only restriction, really, is that all parts except for the last one need to be at least 5MiB.

GCS compose can also have target be one of the source objects, so you can append (and/or prepend) “in place.”

For GCS compose the suffix/prefix need to be separate visible objects (though you can put a lifecycle on them). For multipart, the parts are not really objects ever.

The performance isn't great because because updating the “index” is slow and rate limited, not because the APIs aren't there.

link