I think PolKit can be used to delegate fine-grained control to users. Which other tools are available on Linux to control users' access to fs and network resources? File permissions, systemd-nspawn?
One could write a very small custom binary in C/C++/Rust that runs the command, and then give that binary setuid permissions. That binary can simply run a subprocess, and nothing else.
selinux is the primary one that comes to mind, it definitely ticks the "fine-grained control" checkbox for sure