[cyberpunk]

Pre-Emptive scaling ala erlang can help with scenario one somewhat, if the jobs aren’t locked on some resource. For example, on my erlang system 20 would all run just each slightly slower as they get a smaller amount of scheduler reductions each.

It’s a hard/interesting problem, and harder still once you’re running across a lot of machines — but if they all get slower under the load it turns out that it’s easier to scale / work out a good balance of idle capacity to guarantee x time sla under x requests.

Fast ramp up for additional capacity is important too, but less so if you know to start the process once median execution time drops to some % of your worst target

[eru]

> Pre-Emptive scaling ala erlang can help with scenario one somewhat, if the jobs aren’t locked on some resource. For example, on my erlang system 20 would all run just each slightly slower as they get a smaller amount of scheduler reductions each.

Well, memory is one of these resources that you are often locked on.

If you have enough memory, running 20x the load just goes 20x slower. But if memory is congested, then this can go arbitrarily slower than running your jobs one after another. Eg when you are swapping to a spinning disk.

[cyberpunk]

I’ve never enabled swap on a production system in ~25 years, so i have to say it’s not really my experience.

Shared memory can be a problem, sure, which is why I don’t generally use that either. Erlang, of course, generally does not use shared memory outside of some cases with ETS etc which must be used carefully but i’d rather solve those problems myself.

Concurrency systems in other languages i’ve written, for example Go, there are ways to architect to avoid it also. I’d rather go slightly slower and copy be value than have to solve mutex contention and trying to make everything atomic and so on. YMMV, I don’t work in HPC just large complex busy systems.

[eru]

To make my point more abstractly:

Multiple tasks can share the CPU and just get a bit slower. But if you are out of RAM, you are better off running one thing after another.

Whether you hit swap or OOM was a distraction.