[dekhn]

"""Even with fast NVMe drives and splitting the database into two halves (important data vs. ephemeral data that we could lose on a tmpfs), things got slower and slower. We knew the day would come. The file reached a peak size of 150MB and we were writing it as quickly as the disk I/O would let us. Ain’t that just peachy?"""

Uh, you compressed it first, right? because CPUs can compress data faster than disk I/O.

[bradfitz]

Yeah, I think it was zstd.

But the bigger problem was the curve. Doing something O(N*M) where N (file size) and M (writes per second) were both growing was not a winning strategy, compression or not.

[jeffbee]

Hrmm. Even lz4 level 1 compresses at "only" about 500-1000MB/s on various CPU types, which isn't quiet as fast as NVMe devices demand.

[dekhn]

yes, that's a very recent change (afaict, the big change was going from SATA interface to NVME).