I design sites and this is exactly what is happening. This is also partly to cater for all the different screen sizes, and partly because zooming and DPI are broken beyond repair. W3 broke it first by specifying in CSS 2.1 that DPI is fixed 96dpi, then freedesktop botched HiDPI, and browsers made it even worse by trying to fix it with window.devicePixelRatio and implementing it badly by lying about screen sizes, especially on mobile. And not to mention sites that abuse viewports.
Ctrl +/- does not zoom, it changes the fontsize and devicePixelRatio.
Using a base size (the font size) and scaling everything relative to that using em solves many problems. Sites handle this differently and it is getting progressively worse, confusing screen readers.
In Firefox mobile, when changing the viewport initial-scale the sizes of some elements don't change. Do you know a way to soft-refresh the page so it recalculates all the sizes? This doesn't happen in Chrome mobile.