[emcrazyone]

can anyone chime in as to why use PCIe over something more core to core direct? As I understand it, the CPU still needs to talk to a PCIe host/bridge controller. Why not have something that is more direct between processors?

[sliken]

Hypertransport is an AMD technology that's high bandwidth per line, low latency, and scalable. It's also cache-coherent (well there's a version that is), so it's great for connecting CPUs. But the AMD hardware is flexible and can use the same pins for either.

So the single socket systems can have more pci-e lanes available, but the dual socket has less per socket because some of those lanes are used for hypertransport.

What I can't figure out is why Intel and AMD aren't using similar (Hypertransport for AMD and QPI for intel) to connect directly to GPUs in a cache coherent way. These days the faster interconnects spend a decent fraction of their latency just getting across the PCI-e bus twice.

So 100 Gbit networks, Infiniband, GPUs, etc all could take advantage of a lower latency cache coherent interface, but it's not available.

I suspect mainly because qpi and hypertransport are incompatible and pci-e is good enough for the high volume cases.

[jabl]

Well, AMD is one of the founding members of OpenCAPI, http://opencapi.org/ , so I guess there's some hope. It seems they haven't talked about it wrt Zen/Naples, maybe some later iteration will have it?