Add ipv6 into the mix and it's straight up infeasible to scan for even ONE port on every host!
Port knocking + key auth + non-default port is pretty damn good security, even against zero days in SSH.