The charter is explicitly part of the Canadian constitution. The notwithstanding clause or override is only very rarely invoked (and in many provinces has never been invoked).
It has been used repeatedly, and it's not just provinces either, the federal government can also use it. It's not really rare, it crops up every few years.
Legislation invoking the clause has been enacted only five times across all of Canada. In one of those cases no rights were suspended and invoking it was unnecessary.
Actually, the first invocation was an "Omnibus" invocation, which contained around a dozen separate invocations in itself. Beyond that, no, even 4 invocations isn't small at all. A single invocation is potentially devastating. It's been completely normalized in Quebec, for example.