Because you have to basically implement half a browser for SVG. You need to parse the XML into a DOM and apply CSS that you also need to parse (also javascript in case of animated SVGs that use JS)
SVG specification distinguishes two types of conforming SVG viewers: static and dynamic [1]. The static viewer does not have to support scripting, animations and user interaction.
A good example of a relatively small static SVG viewer/renderer is libRSVG which is used by GNOME project to render icons.
The main size is dealing with libxml, but that's readily handled by smaller xml libraries [http://pugixml.org]. I mean, maybe you don't want to do SVG on an arduino.
Directly comparing SVG and PNG libraries purely based on library size is misleading. PNG's aren't layered, don't contain semantic information, etc. For me librsvg is lightweight since its way better than installing phantomjs as I have been doing to generate embedable PNGs for documents.
A good example of a relatively small static SVG viewer/renderer is libRSVG which is used by GNOME project to render icons.
[1] https://svgwg.org/svg2-draft/conform.html#ConformingSVGViewe...
[2] https://en.wikipedia.org/wiki/Librsvg