Debug mode isn't always needed, though - E.g. C++ with RelWithDebInfo gets you far enough, unless you specifically want to run debugger - I'm one of those who do not use a debugger in their workflow, so a non-issue for me.
That's been my fallback solution too in C++ in the past and it kinda works (and Visual Studio has a special option /Zo since quite a while https://learn.microsoft.com/en-us/cpp/build/reference/zo-enh..., not sure though if that's actually used by cmake's "RelWithDebugInfo" config), but there's still downsides even when not using a debugger. Most importantly: if you have a crash/assert handler which logs out a callstack, that callstack may sometimes be quite misleading (depending on how much the optimiser passes have mangled the compiler output relative to the input source code).