It does not matter that much, relatively speaking. People include 1MB font libraries in their site without a care in the world, and force 1-2MB PNGs and JPEGs down the wire without caring. A few extra bytes in jQuery is nothing compared to that.
However, it does matter in interactive websites because if the js is light it'll load fast and execute fast and you'll have interactivity sooner. It may be the case that images can be loaded later or arent even visible on the home page. And fonts of course will be replaced once they are loaded and the defaults will be used while it loads so it's progressive in a way.
Interactive websites are slow because they are including 1 MB frontend frameworks... and use these bloated modern frameworks to set a class on an element.
Yes, I was just pointing out why having huge fonts/images isnt as big of a blocker as huge JS on today's JS heavy apps with "loading placeholders" and what not. The browser also does font replacement fairly smoothly.
Whether this whole bazaar is ideal or not is a separate issue :)