A manager with an attitude that engineers would pad their estimates to "enjoy the free time" is a guaranteed way to get good engineers to quit a team. Not only does it betray a complete lack of trust, but it means engineers are going to be pressured to lower their estimates, which never turns out well.
Absolutely not. It's hard enough for a seasoned senior engineer to estimate something, let alone a non-technical manager. (Of course, said seasoned engineer might be in a managerial role now but it's still an engineering task.)