[kylegalbraith]

This is a fascinating read. We run a Ceph storage cluster for persisting Docker layer cache [0]. We went from using EBS to Ceph and saw a massive difference in throughput. Went from a write throughput of 146 MB/s and 3,000 IOPS to 900 MB/s and 30,000 IOPS.

The best part is that it pretty much just works. Very little babysitting with the exception of the occasional fs trim or something.

It’s been a massive improvement for our caching system.

[0] https://depot.dev/blog/cache-v2-faster-builds

[guywhocodes]

Did something very similar almost 10 years ago, EBS costs were 10x+ the cost for same perfomance CEPH cluster on the node disks. Eventually we switched to our own racks and cut it almost in ten again. We developed the inhouse expertise for how to do it and we were free.

[e12e]

Did you host ebs on bare metal? How are you hosting ceph - your own/rented metal, ec2 - VMs?

Wasn't immediately clear to me from the blog.

[kylegalbraith]

We started with AWS EBS volumes with BuildKit on EC2. We've now moved to BuildKit on EC2 and a Ceph storage cluster on bare metal EC2 instances.