It's not a password, it is a secondary, single-use recovery second factor.
Carrying that around in a wallet doesn't make you any more vulnerable to physical attackers than carrying your Yubikey on a keyring, and it's much more secure against remote attacks than SMS-2FA (where you can fall victim to SIM swapping, number porting attacks etc).
Lots of people's luggage include enough info to work out their name and likely home location, which is commonly enough to work out their username for a lot of popular services.
That makes the whole "stolen luggage" thing even far riskier. :/