If you live in an "all compiler warnings are errors" world, having two compilers where a bit of code can cause a warning in one compiler and not cause a warning in another compiler is really annoying.
And this is why the intersection area in the Venn diagram of "people that use -Werror" and "people that use multiple compilers" is labeled "people who are making their own lives difficult, for no obvious reason".
Yes, sure - any time different people will be getting different sets of warnings, -Werror (or equivalent) is just going to make your life harder.
You should always strive for zero warnings on all builds across all compilers. It's a realistic, achievable and worthwhile goal. But while pretty much all warnings are worth looking at, a much smaller set are worth breaking the build over. You're much better off (in my view) promoting those particular warnings to errors, and leaving the rest as-is.
Presumably end users who are compiling from source wouldn't want to enable -Werror though.
If you are an application developer and a compiler X gives warnings that compiler Y doesn't for your application, then ideally you should probably fix the warnings from compiler X. An end user probably doesn't care either way, they just want to compile the program without any issues.
Realistically, though, an end user doesn't compile anything these days. If you build Chromium from source you're at least half interested in tinkering with it.
Well certain Linux distros (Gentoo comes to mind) are designed to support their users compiling applications from source. Some users just want to be able to compile using -march=native, and aren't interested in fixing compiler warnings that they didn't introduce. I personally would hate it if I wanted to compile an application from source but -Werror stopped complication through no fault of my own.