[danielheath]

> too big for RAM

64gb ram sticks are about $700 AUD, which brings the problem into range via RAM.

> coming in too fast to write to disk

Good PCIe4 NVMe drives are pretty fast these days. Samsung 980 PRO is under $300 for 1tb, and claims it can write 5gb/sec.

Even if you only get 2 gb/sec, that brings your RAM requirements down to 90gb, which is not an exotic configuration for a high-performance desktop.

I'm eternally astonished by how far hardware has come. Of course, preprocessing the stream on a cheap FPGA would be more elegant, but it's pretty amazing that consumer hardware could do it at all.

[cortesoft]

Ok, then imagine you are getting data at 10/g a second.

[touisteur]

Of from a 200g adapter (or 4). Connect-X 6 EN. Network (LAN at least) almost always has at least 1 order of magnitude over storage... Oh yes I could put in a 10-disk nvme raid array and a liquid cooling system for all this. But then why not compress on the fly if I can anyway...