[inigyou]

You always need a long-lived key somewhere. Keeping it in an HSM is probably the safest, but also pretty expensive.

[DoctorOetker]

Suppose someone could break RSA / Elliptic Curve Crypto, mathematically (because the hardness assumptions were flawed).

This person doesn't need to store the private keys, and has the luxury to recompute them on the fly when needed!

[brendoelfrendo]

Yes, you'll probably need to buy a specific tool or product to have secure key storage. If you're running infrastructure that requires long-lived secrets, then you either need to have a place to keep them and a plan for secrets lifecycle management, or you need to accept the risk that those secrets could be compromised.