Hacker News new | ask | show | jobs
by osti 939 days ago
If I'm not wrong, this is the low latency S3 that is written in Rust. Finally launched after years in the making.
5 comments

S3 Standard has slow first byte latency for three reasons: 1. All data is stored on old school spinning HDDs with multi millisecond seek times 2. There's still Java (and garbage collection) on the request path. There has been a multi-year effort to move the request path entirely to Rust to eliminate GC but Java still remains. 3. To reduce storage costs, objects are erasure-coded "wide", which means many hosts are involved in servicing a request. This means only one such sub-request has to be slow to slow your request down.

The new storage class is SSD-backed, presumably doesn't use Java anywhere, and doesn't stripe your data across as many hosts. It's more expensive because SSDs are more expensive than HDDs and narrow erasure codes are more costly than wide erasure codes.

Source: Used to work on S3.

Minio?
Surely being written in a non-Rust language is not responsible for an extra 40ms of latency, right?

Or is rust really that magic?

Of course not, it's designed differently from the original S3. AWS came out with this to compete with Azure premium blob storage, which has very good first byte latency, and Azure had it 4 years ago..

https://azure.microsoft.com/en-us/blog/premium-block-blob-st...

ShardStore? (More info: https://www.thestack.technology/aws-shardstore-s3/ ) it seems that it was deployed years ago.
Nah, that looks very different, one of the stated goals of S3 Express is to minimize latency, which is the only thing about the Rust S3 that I remember.
Do you have any sources for that? Very interested to know more about this.
Unfortunately I don't, this is already internal information that I don't know if I should say here. I never worked on S3 and I no longer work at AWS so someone from within would have to weigh in.