[joehilton]

Yes this is exactly right: "The physical intepretation of a negative number is the quantified absence of a unit." In the physical world, the absence of a unit just means the perspective doesn't include whatever it is that is intended to be observed, although that thing may exist elsewhere. In other words, a negative number may be an abstraction for quantifying that a unit has left an isolated system, but I'm not sure our current usage of numbers as abstractions captures this - it seems like we often use negative numbers (in any unit) to denote the actual subtraction of that unit universally as opposed to its transfer to another isolated system. (This could be where mathematicians can get into trouble compared to physicists as perhaps they don't have the same habit of observing units.)

[geekishmatt]

Sometimes a subtraction of a unit absolutly legal and you are still in the same model or context. (Temperature loss)

Sometimes a subtraction of a unit is hidden-black-magic and you are switching between two models or contexts but still being in a meta-model or meta-context. (Debits and credits)

It just depends on your model or context how you interprete your value/unit relation, but this is a typical problem in applied physic, engineering or computer science :)