[ledneb]

Assuming serverless is out if the question for your use case, have you tried spending a couple of days investigating a managed Kubernetes cluster with node autoscaling enabled? EKS, AKS, GKE...

Honestly it sounds like you could be at the point where K8s is worthwhile.

[mkopinsky]

I'm considering k8s, but that also means moving services from on-prem to AKS, getting INF to open up the necessary firewall rules to make the services reachable from on-prem, and so on. And as you said, it's definitely days of investigation. I'm not closed to the option.

[p_l]

You might also want to consider one of the simpler pre-packaged solutions for small k8s on premise clusters (like k3s)