That's what I thought reading that too. In general I don't think it's a good idea for maths people to just forget about units ("because it is for physics")
Speaking about math: I'm not even sure what the meaning of polynomial is. E.g. x^2+3x+1. Each component is measured in different units. Is it intuitive?
But such polynomials do turn up in physics, e.g. accelerated motion: x(t) = 1/2 a t^2 + v t + x0. The trick is that the coefficients aren't unitless either.
Such a polynomial only makes sense when x is a dimensionless quantity, or alternatively when the coefficients have appropriate dimensions to compensate.
The definition of a polynomial has nothing to do with units of measurement. The coefficients come from a ring and a ring has nothing g to do with units of measurement.
That’s fine as a pure mathematics perspective, but in physics dimensions come into play often, and one often has polynomials whose domain and coefficients come “tagged” with particular dimensions.
OP did say, “speaking about math...”. The operations on polynomials and dealing with polynomials doesn’t have anything to do with units of measurement. It may be the case that when used in some areas and in some contexts that the units matter but I think it clouds issues to bring them up.
The intuition for operating with polynomials is best obtained by not worrying about units.
In abstract algebra, polynomials are just tuples (a_n, ... a_0) with addition (simple elementwise) and multiplication (more involved) defined. The +s and xs in the "x²+3x+1" notation are just syntax without semantic relevance; in particular x is not a variable and the substitution a ↦ P(a), a ∈ 𝗦 for any a and 𝗦 is not defined. This is because most interesting things about polynomials can be reasoned about without assuming anything about what x is.