[dnaquin]

It's less about speed as it is about size. A general framework is just that, general. You're paying in bytes for stuff you don't necessarily use.

[cookiecaper]

I agree that there's a lot of stuff that doesn't get used in a given site. However, I don't think it would be too difficult to pare it down -- perhaps a step in the compilation process would be parsing your code and removing any functions from the jQuery file that aren't necessary.