My guess would be that the exhaustion the commenter describes comes from the (perceived, and probably accurately so) need to follow certain social norms when in certain social situations. Such social norms wouldn't exist while spending time with one's wife.