[samnco]

I don't know. Maybe the make of the extenders isn't very good, I saw other people with similar issues.

The PSU is the Corsair AX1500i (1500W), with 10x lines for GPUs. It's robust on paper, didn't have any problem with just 4 plugged in.

But I must say... The T630 are very noisy compared to these, but so much more powerful #NotGoingBack

[jacquesm]

I just bought a GTX1080ti + a similar corsair as an upgrade for my 3 year old Dell, it works like a charm.

If you have a PSU that big then that probably isn't the problem. I thought you might be using the PSU that comes with those extender boxes and they usually are very puny (250 W or so).

Do you use it for gaming or for CUDA?

Do you run the 4 GPUs in the extender?

[samnco]

Yes, each GPUs has a 4x -> 16x and a 4x-4x extender, in addition to the m.2 -> PCI-e 4x adapter.

So many potential failure points in there. The sole use case is CUDA. Essentially I wanted a portable cluster with GPUs and that did the work for a couple of month. Now it's getting more serious so the switch to T630 makes sense, and I repurposed the NUCs into the control plane of the K8s cluster.

[jacquesm]

I built this long ago:

https://clustercompute.com/images/image4.jpg

Which was a lot of fun.

Do you have all the GPUs internal to the T630?

Any chance of a picture (of the guts)?

I'm seriously thinking of duplicating your effort.

[samnco]

Here you go :) https://drive.google.com/file/d/0B1CCk51NQ4koSmkxSmxWb1E5Y0E...

Replicating is not very hard. You need a lightweight x86 machine for MAAS, which takes ~20min to install, one VLAN for the iDRAC (IPMI), another for networking that can connect to internet, and off you go. You can also enable KVM power management in MAAS to run the Juju control plane in VMs and save a box if you're limited in compute power.

https://maas.io https://jujucharms.com/docs https://www.ubuntu.com/containers/kubernetes for all the goodies.

If you run into problems, I am SaMnCo on #juju in freenode.

[jacquesm]

Ok, so 2 GPUs in there. Have you tried 4 or is that not possible for some reason?

I have plenty of other hardware floating around here so no problem on hooking it all up.

Thank you for the image.

[samnco]

I have not, but it is technically possible. the PSU is the double 1100W with the GPU enablement kit. Up to 4x PCI-x 16x full speed. Also up to 1.5TB RAM, and 8x 3.5" HDD or 16x 2.5". I didn't go this far though ($$$...)