There's a significant niche for such features, we're dozens. That's the reason why people like stuff like Zig. Pointer arithmetic isn't just dangerous, it's cool too.
I'm not saying that memory unsafety is useless or even undesirable, I think the "problem" is that it's the default in "current popular and actually used high-performance languages", which means that stuff that needs to be fast and secure (like web browsers) have a hard time.
Maybe my "most of the time" was too strong and I ignored big domains that I don't know. But for the part that I relatively know (web stuff mostly), safety and speed together matters a lot. The thing is, since unsafety tends to be viral (an unsafe part of your stack can compromise everything), people get very paranoid about "unsafe languages".
I understand your point, but for the most part memory-safety features and speed of execution run contrary to one another.
Garbage collection and bounds checking is big sticking point for systems / high performance programming, and to an extent real-time programming.
As far as I can tell, Rust should be about the limit of what's possible if you want to have your cake and eat it too. But I'll doubt it'll ever replace C for speed-critical applications.