That’s a very toxic attitude to have, especially as a manager. You don’t know that “nobody gets offended”. All you know is that “no one chose to express their discomfort”, a very different thing.
It's pretty easy to make sure nobody gets offended - you communicate (in a serious setting) that you value results over hours worked and you demonstrate this in practice. At that point irony and sarcasm is safe territory (appropriate or tasteful is another matter). Being afraid to joke about this because it could be missunderstood tells me you are failing at that which is far more important than worrying about what are potential interpretations of your jokes.
It’s not being afraid to joke, it’s making sure that employees do not feel like they are receiving mixed signals (“he says results are what matters but he’s always joking about me leaving early when I’m leaving at a reasonable hour and getting my work done?”). This is particularly important when considering employees who might not have English as their first language, or come from a culture where joking is perceived differently.
I would say this applies to any value.