Im not sure that real in the context of math means anything other than the real numbers or (at a stretch) existence. I agree with you.
And no one talks about this but a priori i dont see why even the natural numbers have anything to do with reality either. "Counting" some objects seemingly would require a notion of the object and some way to determine whether each part of reality was or was not the object which already seems fairly abstract and removed from the world. I think certain parts of math just seem more intuitive to certain people