Vim is my main editor, my main gripe is that it performs terribly (without any plugins) with either relative numbers, cursor line or syntax highlighting in files larger than 200 lines.
Did you try different terminal emulators? All these features sound like they cause a lot of redraws, so it could be that the terminal emulator's renderer is the bottleneck here.