Through many experiences I think you're right. However that is an issue that could be solved by being better at measuring productivity.
If I can produce better results while playing games for 4 hours and working for 4 hours in a closed office than I could while "working diligently" for 8 hours in an open office, then it's hard to argue for the value of visually supervising someone.