You think that a page that loads jQuery but doesn't use most of it incurs a performance penalty? I'll grant there are a few initializations and dom checks that jQuery does, but it probably blocks for all of 20ms.
Most modern browsers are able to paint while loading script files, especially if you put your script files at the bottom of the page, where they belong. So, no, 20ms of parsing, compiling, and a small amount of execution is not a lot of time. Period.