[dded]

Is this a hard problem for kernel devs to solve? Or could it be as simple as using a timeout when the kernel's OOM killer is disabled? That is, when a cgroup is at the limit, the kernel waits for some finite amount of time and then starts killing things itself. Or would this cause other problems?

[npsimons]

The oom-killer is the kernel devs solution; if you're disabling it, you should know better, and more importantly, you're on your own. Also, the kind of people who would disable the oom-killer are probably the same kind who wouldn't want the kernel messing around with their cgroup; in many senses, that just looks like another oom-killer, only tuned to wait a bit longer.