It's not just raw input speed - writing on paper allows you to quickly jump around the text, making small corrections, adding notes/symbols/drawings, etc.
I can jump around and make corrections in vim far quicker than I can flipping a pencil, erasing, flipping it again, writing correction. And that's for a simple edit that doesn't require moving block of text. Hand writing it slow and inefficient.
Because you don't write a lot of code on paper? Working with text (for notes or prose) is very different than working with code.
You don't think "I need to correct that thing located 23 words forward so I use 23w or press w 23 times" - you position your hand and make a correction.