Throughout history, big advances have come from humans having more "idle time", so we should be aiming for the population to be less busy as they can then hopefully focus on pursuing the arts or sciences.
Well, wars are going to happen anyway. If we abolish all idle time, it's pretty much the same as getting rid of artists, poets, philosophers, writers etc.
I agree with you on principle, but I don't think it's straightforward as your point states.