I'm writing this on a dual E5-2670 v1 system I built for about 1000USD (not including monitors). 16C/32T, intell s2600cp motherboard, 48GB ECC, 256GB SSD (some cheap one), dual 3TB WD Red. Running docker and kvm on the host, and also currently using it as my main desktop. I have a freenas guest with ZFS handling the HDDs (PCI passthrough of a SATA controller), and a Windows 10 guest with GPU passthrough for office and games. All-in-all it's been a fun project, especially for the price. Incredible how cheap hardware can be these days. I paid 60USD for the CPUs when I did my build. I think they've gone up about 50% since then, but are still a steal in my opinion.
Just out of curiosity, do you have an idea how much power that system uses?
I will probably get myself a new desktop computer sometime this year, and those specs sound pretty sweet. But I want to keep an eye on power consumption, too, and I don't my desktop to either melt or have its fans create a tornado in my living room... ;-)
I went ahead and rebooted it so I could plug my Kill-A-Watt back in just for you ;). It idles at right around 110W. I did a quick sysbench to peg all the cores and it goes up to about 300W. Note that I have some other things sipping from the meter so it's actually several watts lower for just the tower. With the Windows 10 guest running (host using GTX 950, windows RX 460) it idles between 190 and 250.
BTW I couldn't really tell any difference in the fan noise levels when the CPUs were pegged, though the temps went from ~40C to ~60C. It's already pretty quiet. Just barely above the level where I notice it.
I'm really interested in this too. I'm tossing up between something like this (maximum thread count but probably high power usage) or a Ryzen build (simple and low power).
That CPU has a TDP of 115W but I can't find much information on the idle power usage.
I would also like to see a write-up of the GPU pass-through setup since that's something I've been wanting to have on my local system for ages, i.e. vm host system => dev vm 1 ... vm N, Windows VM + dedicated GPU for gaming etc.
If I were doing a build today, I would be sorely tempted by Ryzen, mostly because it would be new hardware and the AM4 platform. That said, having a server board has turned out to be really nice for a lot of things, such as ECC memory and nice virtualization features. It looks like Ryzen has some issues with kvm passthrough so far[0]. My favorite resources are [1] and [2].
What is your host OS / hyperviser? I would like to make similar setup and was wondering if Ubuntu as main host would allow the pci passthrough and all..
I'm using Arch with kvm/libvirt. Most of the configuration of guests is done via virt-manager. Since a lot of this is moving relatively fast, I recommend using some sort of a rolling distro. I believe the libvirt devs mostly use Fedora.
As you are mentioning that you custom built the system and the intel s2600cp motherboard comes in the EE-ATX ( Enhanced Extended ATX ) format, can you please specify the computer case you are using ? The official ones ( Supermicro, etc ) are expensive if bought new, same for used ones on ebay as they are large and heavy, thus the shipping fee is killing the deal.
My HP z820 uses almost 300w at load and some fan noise with dual E5-2660s, 128GB RAM, and an nVidia Quadro K2000. I've heard that some people that do audio/video with similar rigs replaced the factory CPU coolers and fans with thermaltake products for quieter rigs. It noticeable but not what I would call annoying. It is louder than the 2010 Mac mini it replaces when at normal load but way quieter than when the mini blows at full speed during swapping or intensive going usage.
I use a 2x E5-2665 v2 machine (S2600CP4) for building stuff. It's in very small, very cramped 2S Supermicro case (2U, less than 60 cm deep). Fun machine. ing loud as hell, uses stacked 40 mm fans... :)
HP z820/z620/z420 and Dell T7600/T5600 are great used workstations that support the Xeon E5-26xx V1 series processors at very reasonable prices on eBay. Workstations are $300-$700 (maybe add CPU, RAM, HD, graphics card) shipped. I've bought a z820 and z620 at $420 & $499 respectively. My machines were partially usable missing only a second CPU & graphics card in the z820. z620 was fully usable with dual E5-2620s. I added a pair of E5-2660s, 128GB of RAM, and a second CPU cooler+fan to the z820 for $500 shipped and swapped the graphics card. So for $920, I got a 32 threads and 128GB beast z820 (probably would have cost $20K new) and a E5-2640 and 16GB of RAM to sell. So easy to work with too with their tool-less design.
It's a great 4K video/photo editing & OpenShift lab machine. Literally 300 chrome tabs and it hasn't cracked past 16GB of RAM usage in RHEL. Just need to spin up some VMs now.
I am using the Fractal Design Define XL R2 (what a name) for my Supermicro dual CPU mobo, I drilled one extra mounting hole and the mobo mounted just fine.
To the extent that dual socket LGA2011 ATX/LLB boards moved up in price to a point where you could pay three or four times more for a suitable mobo than for sixteen cores of CPU.