You can capture straight from the GPU into the GPU's video encoder and directly get h265 frames efficiently without laundering the whole framebuffer through system RAM each time.
Nvenc is actually quite nice and easy to use for that, but if you want to do any serious post processing you'll need to parse h265 (headers at least) and it's not the easiest parser to write.
On Windows, there's an API to capture the display as a DX texture. Then either use NVENC or AMF to encode. You'll get compressed frames that you can just stuff straight into whichever video container you like.