[franciscrick1]

Pieces of paper work pretty well.

Edit: literally any text data storage medium that can be powered down and locked in a safe will work perfectly well for any of this. It doesn't matter if it's a flash drive, floppy disk, CD ROM, piece of paper, or sheet of gold with characters scratched into it. Literally anything will work. You do not have to buy these $50 glorified flash drives to store your bitcoin.

[nybble41]

That takes care of offline key storage, but you'll need a lot more than just a piece of paper (and a safe) to actually use the key securely: a secure offline computer, for a start, and a process for transferring the key and transaction data into that computer, signing the transaction, and getting the signed transaction back to a connected system for upload. At which point you've basically re-invented an ad-hoc, inefficient, and quite likely insecure hardware wallet / HSM.

Or were you just planning to scan the private key QR code with your (compromised) smartphone?

HSMs are a lot more than "glorified flash drives". The most important difference is that they keep the key data securely stored on the HSM and only allow it to be used in specific ways, such as signing individual transactions. Depending on the HSM you may even be able to program it to only sign transactions which meet specific requirements, such as transfers from cold storage to known hot wallet addresses.