Yeah, I can't relate to that at all. My Matter devices are more reliable than what they replaced (mostly switches and dimmers so far), and HomeKit support has "just worked". Matter's a 16-month-old baby, though, so YMMV.
Home Assistant: The State of Matter (Jan, 2024)
"We believe in Matter: it’s open source, and most importantly, it’s fully local by default...
Matter launched only one year ago, compared to twenty years for a standard like Zigbee, so it’s important to account for that in your expectations."