Antimatter isn't some unobserved theoretical thing. It's produced daily at most large hospitals. We absolutely know that its electric charge is opposite that of the normal matter counterpart.
How do we know that? Because of how it behaves in electric and magnetic fields. In particular, how it accelerates. It obviously has a negative sign in it somewhere. My point is that we don't know if that sign flip is in charge or mass. A lot of the math behaves in accordance with observation whether we flip the charge or the mass.
Here is a contrived example calculating a hypothetical quantity X = m * Y.
Suppose we observe that X is always negative.
By your logic, we would then assume that Y is always negative.
This is true if m is never negative, but it is somewhat possible that we would eventually find a situation where m is negative and Y is positive.