In my limited experience, rendering vector fonts is among the slowest parts of rendering text in a terminal. The difference between an xterm with a TrueType or OpenType font and an xterm with a PCF is noticeable.
This can make a huge difference. It can easily be worth the performance benefit to find a well-designed bitmap font[1].
I also highly recommend using urxvt[2], which not only has very fast rendering and support for a lot of modern features, it has a few settings you enable that specifically address the too-much-scrolling speed issue. From urxvt(1):
jumpScroll: boolean
[...] specify that jump scrolling should be used. When receiving
lots of lines, urxvt will only scroll once a whole screen height
of lines has been read, resulting in fewer updates while still
displaying every received line [...]
skipScroll: boolean
[...] specify that skip scrolling should be used. When receiving
lots of lines, urxvt will only scroll once in a while (around 60
times per second), resulting in far fewer updates. This can result
in urxvt not ever displaying some of the lines it receives [...]
These options can completely fix the scrolling lag problem.
I also highly recommend using urxvt[2], which not only has very fast rendering and support for a lot of modern features, it has a few settings you enable that specifically address the too-much-scrolling speed issue. From urxvt(1):
These options can completely fix the scrolling lag problem.[1] e.g. http://terminus-font.sourceforge.net/
[2] http://software.schmorp.de/pkg/rxvt-unicode.html