[api]

I’ve wondered for a while: given the lower cost of SSD per GB could you build a very wide RAID0 style striped array of SSDs (maybe one per slot) to get almost RAM like read speeds?

To really go fast you’d probably have to do PCB layout and do like 256 or 1024 chips in parallel with a fast SRAM aggregation buffer feeding a GPU or TPU rig.

Or could you do the same with custom layout of cheap slower RAM?

I wonder if anyone is doing this? You would flash in a model and then just run it. It would need RAM for context but much less of it.

[colingauvin]

The problem is you'd have to traverse the PCI-E bus every transfer. Even with DMA it still has to physically get off the drive and onto the card. Even with the mythical PCI-E 6, if you had enough NVMEs to saturate, you have to do two transfers to get it to the inference hardware. And that tops out at 128 GB/s which is roughly the speed of DDR5 but with one extra hop.

Optane would actually be useful in this era. Intel was ahead of their time.