Sure! There are a lot of ways you can do this. One is to use PIV mode, which exposes a PKCS#11 interface that can be used by OpenSSL. Yubico has some nice docs on using a Yubikey as an HSM to store CA keys: https://developers.yubico.com/yubico-piv-tool/Certificate_Au...
Note that (at least in the method described in that document), the Yubikey does not _generate_ the private key, it merely stores it securely. You need to generate it on a computer, which should preferably be a brand new laptop, airgapped, purchased from a physical store and not online (continue to layer defenses up to your desired level of paranoia).
Actually, in PIV mode it's my understanding that you can have the Yubikey generate the private key if you want it to.
Note that the key is intended to be difficult to extract from the device and there is not intended to be an interface for doing so.
If you are using it with RSA key, you need to decide if you trust that it is generating the key properly. Keep in mind Yubikeys are built on GlobalPlatform/JavaCard which is an extremely high value target. There would definitely be the economic incentive for a well-funded adversary to backdoor the RNG system.
If you are using a Yubikey with EC cryptography, the security analysis gets even more complicated. The curve used need to be a safe one.
And, if the RNG has been tampered with or incorrectly designed, EC signatures can actually leak the private key.
Which is to say that if there is a flaw in the Yubikey, JavaCard, GlobalPlatform, or the specific chips they are using, and you are either (A) having it generate the private RSA keys, or (B) using it with elliptic curves, then there is the potential that the device is not at all secure.
Note that backdooring crypto-specific chips is a thing. TPM and other special-purpose crypto chips have been discovered to have been both backdoored and vulnerable to implementation issues.
Which is completely unacceptable - "you had one job".
You might want to look at the Nitrokey HSM. I'm using one for a project and I was blown away by the feature set for the cost (as compared with the SafeNet devices I originally looked at). Its signing throughput is too slow to run a website with any significant new connection volume, but for personal applications it could be made to work. As just an HSM I think they're a phenomenal value.
There is also the non-insignificant issue of the $500 price tag. Wouldn't bat an eyelid if this was for professional/corporate use, but $500 out of my own pocket for my personal home server is a bit steep.
Note that (at least in the method described in that document), the Yubikey does not _generate_ the private key, it merely stores it securely. You need to generate it on a computer, which should preferably be a brand new laptop, airgapped, purchased from a physical store and not online (continue to layer defenses up to your desired level of paranoia).