Well if you're uncomfortable with its priviged daemon, you can always switch to CRI-O with Red Hat tooling for it. But for all my years with Docker as the container runtime, all security related problems have occured within the backend code, not Docker, not Linux cgroups, not Linux itself.
I've worked with some big customers in the financial industry, and this is exactly what we do. Podman implements the same CLI as docker, so you can basically just `s/docker/podman/g` (as long as you don't use docker-compose).
It's also a lot easier to debug and see what's happening without that daemon sitting in the middle of all the traditional linux tools.
containers in general are horrible wrt security because they are architecturally flawed - they pretend to have some sort of 'isolation' but that was crap docker marketing people just made up - there is no isolation - k8s pushes this agenda further by declaring that multi-tenant workloads are perfectly normal and ok for containers which they absolutely are not
just look at the CVEs from recent years:
* docker doomsday
* escaping like a rkt
* cryptojacking? - that didn't even exist until containers were here!