You still need access to the host, via ssh for example, to start the containers and do some basic maintenance. Won't you have fail2ban installed on the host since your ssh port would be open?
If you need direct access to the host, it’s probably a non production environment or you’re doing containers wrong. Kubernetes clusters provisioned with Terraform, for example, should almost never require ssh access to workers nodes.