[frumiousirc]

In particle physics, processing jobs face a limit of about 2GB/core. This is driven by the realities of "grid computing". The hardware in the computing facilities have approximately 2GB/core and jobs allocate based on number of core. This ratio has been fairly stable for a long time. Some facilities may have a bit more RAM per CPU core but 2GB/core is the general rule of thumb. I hazard a guess that this will not change throughout the (hopefully small number of) years that this memory cost bubble persists. In fact, the increase in cost may lead some facilities to keep older hardware going beyond its EOL/warranty period, prolonging a ratio that may actually be relatively generous compared to today's hardware systems.

As a consequence of this fixed RAM/core ratio, substantial software development effort goes to either making jobs fit in 2GB or if that is not possible then to utilize multithreading. Generally, particle physics processing does not particularly benefit from MT except in this fixed RAM/core situation. Sometimes large memory jobs are needed (inherently or because of bloat that is too costly to improve). When run on the "grid", these jobs must allocate multiple cores just to get "their" memory. If those jobs can use the extra cores, overall throughput does not have to suffer.

That's for conventional software, which still makes up the bulk of the computing. The situation for the growing amount of GPU-accelerated software is different and more varied. One trend can be seen relating to VRAM. Research groups with easy access to big GPUs like A100 write code to fit or exceed the relatively copious VRAM limits of the data-center GPUs, while groups that lack easy access to DC GPUs but have access to more modest "gamer" GPUs write more advanced software that can fit the smaller VRAM. In some cases, they write the software so it can scale the computation, keeping GPU utilization high while staying just under the VRAM limit.

General budget crisis and limited resources in the particle physics field are in part responsible for all of this tailoring of the software to fit the hardware. If better funded, particle physicists could spend more time doing physics and less time squeezing last drops of processing power.

[jcgrillo]

I remember reading somewhere about an HPC "rule of thumb" that was like 1GB RAM/1GHz of processing is about the right amount for most applications.. Don't recall where that came from or what the exact reasoning was. That kind of aligns with 2GB/core.. Was that a thing?

[frumiousirc]

Maybe it was a thing from before clock speeds saturated and Moore's law mutated into being satisfied by ever increasing core count? It could have been useful when spec'ing hardware with the goal of running multi-threaded jobs that scale both their throughput and RAM requirements linearly with number of threads. That would make it sort of a continuous version of the more discrete GB/core rule of thumb. However, such linearly-scaled MT jobs are a subset of all MT jobs. And, ST jobs are not covered. A 2GB ST job on a 1GB / 1GHz machine will again need two cores, wasting one. When run on a 5GB/5GHz machine the job needs only its one core but must waste 3GB.

In particle physics, as more of the code is being GPU-accelerated, there is now another integer ratio to worry about optimizing: CPU core per GPU device. Across the landscape, some jobs have zero GPU acceleration, others may need 100 cores to keep a GPU busy, or only 1 core. Yet others can tune their CPU/GPU ratio to optimize throughput given what hardware ratio a given facility provides. Only a fraction of the software in the ecosystem takes up this challenge.

Most physicist pretending to be software developers or vice versa who are involved in the field do not consider any of these computing realities. At some level, that's natural and excusable. It's hard enough to develop the simulation and reconstruction and analysis algorithms. Simultaneously optimizing their implementation for throughput on a given hardware assumption is even harder. Harder still is to do that optimization over the variety of hardware assumptions. There are only a few cases where this holistic thinking has driven the design of the software.

[jcgrillo]

It's endlessly fascinating to me how long it's taking for GPUs to be adopted for this kind of workload. I remember all the excitement around the GPGPU era, OpenCL, and eventually CUDA. That was like 15yr ago! Yes, GPUs can do a fantastic amount of computing. But it's really hard to make them do it efficiently. I think maybe the implicit assumption at the time was that something would come along that would make it easier. Despite continuous advances over the last decade+ it's still really hard.

I feel like we're about to learn a similar lesson with generative AI. Things don't always get easier/better/faster.