Almost back to point A then. If the server is compromised in some way, they can use (not copy) OP's keys and use them to clone repos/inject code/etc..