[CHY872]

The other element which appears unsaid is that in a typical datacentre, your bisection bandwidth is typically << (num computers * network bandwidth per computer). Or in other words, even if your computation is bandwidth and not latency starved, you're not obviously going to be able to do a gigantic data shuffle quickly unless it's within a rack. This is to say, I'm not sure _just how much_ this would affect your overall target system architecture at present. Once you're switching at a couple of terabits things are likely quite different in those terms.

The other element which is a bit scary is that it's fairly rare these days for mass market companies to index deeply on tech which isn't available in public clouds, so until AWS supports this sort of thing, it's unlikely that many folks will target it. You can kind of see this with the Optane PDIMMs - they looked absolutely fantastic, but given you couldn't get them on any AWS instance there wasn't much point actually trying to use them outside of very specific applications - as a software engineer this hardware lets me build my software very differently and in a simpler way, but how can I possibly risk architecting based on that if it then cannot support a customer's cloud migration?

Obviously v different in HPC contexts.

And it should always be said - latency is very different between these. Memory latency still measured in nanoseconds, PCIe latency still measured in 10s of microseconds, about 3 orders of magnitude difference.

[touisteur]

I can see many reasons why Optane PDIMMs didn't take off, other than not being cloud-available (which Intel might have subsidized with the help of a big software player and a series of killer apps). Price was excruciating, number of write cycles (so lifetime of machine, architecture) not very clear and upgrade path / exit hatch was also not clear. Also the programming model wasn't very clear and not many big important applications had migrated to use them efficiently or in an interesting manner (we were still in the 'crazy interesting papers' cycle).

It still grates me that the only interface we ended up with for high speed durable data is nvme through pcie and regret the lost promise of fast byte-addressable persistent memory, but once again, worse is better seems to have won?

[wtallis]

> number of write cycles (so lifetime of machine, architecture) not very clear

I thought Intel was pretty clear by saying they warranted the modules for five years of continuous operation regardless of workload.

[CHY872]

right - why won't 'networking w/ 800GbE' be different (in terms of weird programming model, crazy interesting papers, etc)?

[touisteur]

Very good question. Being up to my eyeballs on 400G, dpdk, gpudirect, spdk right now I see a way forward, but also I don't see how we 'progress' more without them, where's the 'other' path? Integrated chips and like Apple's m1/2 or NVIDIA's GH and AMD's MI300X?