[cjbgkagh]

But does the data fit in compute? Assuming computation needs to be done on the data you could quickly run out of compute power on a single machine. Especially since cost of CPU power is super linear with some pretty hard limits.

[adgjlsfhk1]

You can get 256 cores of Zen4 (with 8 GPUs) in a box. That's a lot of compute. You definitely can be compute constrained but if you are writing efficient code you can do a lot.

[cjbgkagh]

That's $44K in CPUs alone, and these clock slower than desktop cores so it's not exactly equivalent. There are many tasks for which GPUs are not relevant. I have an expensive compute limited task and for my purposes a mini cluster of desktop CPUs still seems to be the way to go.

[adgjlsfhk1]

Lets say that the cluster of desktop CPUs is 10 CPUs with 8 cores each for 80 cores at 4ghz in total. Instead you can get a server with 64 cores at 2.5 GHz and 1TB of memory for $50k from Dell (adding another 64 cores would be about $6000 more). While this may sound like a lot less performance, you won't be wasting a ton of performance reading data from disk and communicating over a network All of your cores share the same RAM and cache so your computation will be a ton faster. Furthermore, your cores will all be running the same instruction set so you can take more advantage of vectorization). If you aren't using compute frequently enough to justify a server like this, you can get that level of performance and memory from AWS for $4 an hour.

[cjbgkagh]

My task has next to no network or hdd. Data parallel and the task is mostly branches, so GPUs are not effective either. And why use 8 core desktops when 16 core desktops exist. 10 desktops at $1.5K each is way cheaper. It’s also the kind of data that can go into the cloud. Not saying it’s the typical workload but it is my workload. Also it’s CPU bound so they go 100% all the time.

[cjbgkagh]

Edit: cannot go into the cloud…

[jacquesm]

And if your workload is FP and nicely parallelizes and fits in 12 GB or less than you might be able to get that kind of performance for the price of a high end GPU + a box to stick it in. GPUs are insanely cost effective for problems that match their geometry. So much so that it can be worth thinking about it for a little while to see if you can make it fit.