I don't think this is the case. Losing either loses the server in the sense that you won't be able to access it. The fix is that the "something you have" is both your phone and emergency codes.
In security, losing the server doesn't mean losing access to the server. Losing the server means that someone else has access to the server, as in an adversary.
Yes, if you lose either factor, you can't access the server. This is why with Google Authenticator you also get a one time pad with emergency codes. However, I don't know how well this would work with two-factor SSH... you'd need a separate one time pad for each server. And I'm not sure how the Google PAM module handles emergency codes.
Yes, if you lose either factor, you can't access the server. This is why with Google Authenticator you also get a one time pad with emergency codes. However, I don't know how well this would work with two-factor SSH... you'd need a separate one time pad for each server. And I'm not sure how the Google PAM module handles emergency codes.