In part especially since KDE deliberately breaks up their core libraries into multiple sub-libraries(https://develop.kde.org/products/frameworks/), in order to make them available for reuse by other Qt projects. Which is not the case for XFCE. KDE also depends on Qt which is similarly broken up into smaller packages(e.g. https://packages.debian.org/buster/libqt5gui5 is depended on by https://packages.debian.org/buster/libkf5auth5). Even though the actual attack surface in terms of exposed functionality is similar.