I'd very much appreciate it if both of you present some evidence, because I now realize my vague assumptions about this are not really based on anything...
The general phenomenon here is very well known; just searching around for this I found stuff like "UK time use data for the period 1961-2001 do indeed indicate a reversal of the previously negative leisure/status gradient". ( https://www.iser.essex.ac.uk/files/iser_working_papers/2005-... ). But that paper is obviously more concerned with the UK than the US.
Data on this is directly available from the American Time Use Survey, if you want to tabulate it yourself -- while they do collect various data on employment status, they don't publish a summary of time use by employment status.
It's weird to assume that all time spent outside a paying job is leisure time. People who have money can afford to spend a lot less time cooking, taking care of children and elderly parents, etc.