"You give" implies the ability to decide that. The problem is that there's always the threat of people such as slavers, religious fanatics, ethnonationalists etc coming to power. The more that power is centralized, the more they can do with it.
And yes, conversely, when power is decentralized, you don't get "benevolent dictators" who might otherwise do some good. My point is that this argument doesn't really work if applied consistently in all cases - or, at least, not without more digging into the specifics of costs vs benefits. Personally, I'm not convinced that centralization is a net good overall, if only because it makes large standing armies possible.