And this is why the intersection area in the Venn diagram of "people that use -Werror" and "people that use multiple compilers" is labeled "people who are making their own lives difficult, for no obvious reason".
Yes, sure - any time different people will be getting different sets of warnings, -Werror (or equivalent) is just going to make your life harder.
You should always strive for zero warnings on all builds across all compilers. It's a realistic, achievable and worthwhile goal. But while pretty much all warnings are worth looking at, a much smaller set are worth breaking the build over. You're much better off (in my view) promoting those particular warnings to errors, and leaving the rest as-is.