[mthoms]

That's quite an assumption (that it has anything to do with physical appearance).

It's equally possible that it was intended as a shorthand for "cheer up, it's not that bad" or "try to keep your chin up". Both of which actually indicate that the speaker cares about their colleague.

Sure, I can see contexts where it would be inappropriate to say but that doesn't make it always inappropriate to say.