[teractiveodular]

For comparison, while GCS doesn't support appends directly, there's hacky but effective workaround in that you can compose existing objects together into new objects, without having to read & write the data. If you have existing object A, upload new object B, and compose A and B together so that the resulting object is also called A, this effectively functions the same as appending B into A.

https://cloud.google.com/storage/docs/composite-objects#appe...

[pclmulqdq]

Colossus allows appends, so it would make sense that there's a backdoor way to take advantage of that in GCS. It seemed silly of me that Google didn't just directly allow appends given the architecture.

[CharlieDigital]

There are some limitations[0] to work around (can only compose 32 at a time and it doesn't auto delete composed parts), but I find this approach super useful for data ingest and ETL processing flows while being quite easy to use.

[0] https://chrlschn.dev/blog/2024/07/merging-objects-in-google-...

[vdm]

S3 can also do this https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPa...

[thinkharderdev]

It's similar but no really the same thing. It has to be done up front by initiating a multi-part upload to start. The parts are still technically accessible as S3 objects but through a different API. But the biggest limitation is that each part has to be >5MB (except for the final part)

[new_user_final]

It's totally different thing and requires special way to initiate multi-part uploading.

[vdm]

totally different how?

[Epicism]

More constraints?