We're still in the early days of risc-v availability. Arm has been widely deployed and the compilers optimised by both individuals and companies. I expect this comparison to look better in a year (not better than rpi4, just a smaller gap)
Fortunately, at least for video codecs, there's a hardware acceleration block, and it is significantly stronger than the one rpi4's SoC has.
Unfortunately, there's no drivers in the current images, and it'll take time to have proper open drivers and integration with common APIs such as vdpau.