Unless you have exceptionally good controls it's very hard to be sure there is not an SSH key sitting on some machine that would allow access and possible nefarious activity by a dishonest ex-administrator
This is one of many great reasons to rotate them regularly in an automated way. e.g. https://derpops.bike/2014/06/07/ssh-key-rotation-with-ansibl... or update it in your master image / wherever it comes from if doing immutable system images for deployments.
edit: also, use a bastion host which has the keys on it and don't allow them to be removed / used from laptops directly.
Let your Puppet/Ansible clear out all non-managed keys. If it's not in version control, you don't know who did what when. That's a nightmare as soon as you are more than two admins.
Also, the CA mode of OpenSSH is great. More people should use it. It's like PKI but sane.
edit: also, use a bastion host which has the keys on it and don't allow them to be removed / used from laptops directly.