The problem is that not all humans are decent. The more of them you hire, the more likely one of them is the asshole who will take advantage of flexibility and ruin it for the rest.
I'm not sure I agree with this. Do you think an otherwise good person/employee with a newfound ability to work remote would become an asshole and abuse the privilege? Seems like they'd probably be an asshole to begin with. Also, if someone is an asshole and taking advantage of the flexibility, why not fire them? I'm not sure how keeping them in an office solves this issue of people being bad employees.
The 'one bad apple' thing never cut it for me. If you are bad apple's manager, you have measurable metrics on her performance, and you act based on that, no more no less. This has no bearing on any of the other apples.
And what exactly do you think objective metrics for developers should be? LoC? Defects? As soon as a measure becomes a target, it ceases to be a useful measure.
If one person in a group takes unfair advantage of flexibility by e.g. only going to the office half as much as the rest, it sure will have an effect on the "other apples".
Why would she need to go to the office? Work is something you do, not a place your'e at. If you, as a manager do not have any way of measuring how the people you are managing, then you have no reason to be there.