With containers, you usually get incompatible dynamic loaders in the containers (see mananaysiempre' comment; the glibc dynamic linker sees rather active development in some LTS distributions). This wouldn't be possible if the loader were part of the kernel.
Non-ELF loaders are fairly common, too. It's how Wine works, and how Microsoft reuses PE/COFF SQL Server binaries on Linux.
There's also binfmt support, which can check a supposedly executable file against some magic and auto-launch an interpreter (like wine or java or dosemu). I looked into it for something once but in my case the magic wasn't good enough.
Part of it is the Glibc loader’s carnal knowledge of Glibc proper; there’s essentially no module boundary there. (That’s not completely unjustified, but Glibc is especially hostile there, like in its many other architectural choices.) Musl outright merges the two into a single binary. So if you want to do a loader then you’re also doing a libc.
Part of it for desktop Linux specifically is that a lot of the graphics stack is very unfriendly to alternative libcs or loaders. For example, Wayland is nominally a protocol admitting multiple implementations, but if you want to not be dumb[1] and do GPU-accelerated graphics, then the ABI ties you to libwayland.so specifically (event-loop opinions and all) in order to load vendor-specific userspace drivers, which entails your distro’s preferred libc (probably Glibc).
[1] There can of course be good engineering reasons to be dumb.
Why do you need "vendor-specific userspace drivers"? I thought graphic acceleration uses OpenGL/Vulkan, and non-accelerated graphics uses DRM? And there are no "drivers" for Wayland compositors?
It's hard to describe how complex this stuff is. Shared object loaders are essentially primitive package managers, topologically sortinf dependencies and everything...
Non-ELF loaders are fairly common, too. It's how Wine works, and how Microsoft reuses PE/COFF SQL Server binaries on Linux.