I wrote a standalone CardDAV server ages ago and the biggest frustration for me was just how buggy the clients were. At some point I stopped self-hosting and moved on.
I've been self-hosting one of the CalDAV+CardDAV servers based on Sabre for few years now and that thing is solid. Multiple clients - iPhone, Android, Thunderbird. Using it with very few users though.
This would've been years ago. Both MacOS and iOS were insanely buggy (Tiger era). I think the lack of momentum is due to the fact that CardDAV is pretty darn simple, and CalDAV is… idk. Complex yet mature?