[netik]

A user can always start their own SSH server. Just because you've decided to move it to a different port doesn't really encourage them. I suppose you could make this a bit more difficult for them by removing compilers (no really, you don't need compilers everywhere) and making sshd owned by root, mode 700...

However, proper ingress filtering or local iptables/pf rules would stop any unwanted inbound traffic from reaching your server, and you should definitely be using ingress and egress filtering on your network.

[TheDong]

removing firewalls has effectively no benefit; a non-root user can trivially download and run an arbitrary distro or package manager (e.g. nix from nixos, portage from a gentoo prefix, etc) and effectively do a chroot + package management without root.

[JasuM]

The point was that you could more easily replace the SSH server with a malicious one and e.g. hijack your agent when you connect to it.