[kingo55]

This is the sole reason I'm avoiding passkey setup. How big of a worry is it?

[CharlesW]

Assuming you've only set one passkey (you can add multiple passkeys from different passkey managers) it's no more of an inconvenience than if you lost your password. Normal recovery/backup code processes still work.