Well, they create a culture of distrust, and when crossed with issues like drug use which may not be relevant to work concerns, it's going to scare high-value prospective employees.
Some kinds of organizations require some level of distrust in their culture. And the issue seems to really lie with how the employer handles past drug use. My understanding is many organizations, including some government agencies, tend to be fairly lenient about admitted past drug use (depending on many factors like how long ago it was and the kind of drug).
I think in an ideal world, questions about past use would never be asked and drug tests would be strictly limited to determining if someone is high/drunk on the job. The fundamental problem is employers losing potentially valuable employees due to their unnecessary and invasive drug policies. The polygraph isn't to blame.
If a culture exists which relies upon distrust, that culture is tightly-controlled, so that such distrust can be usefully exploited by the /elite/ to put the community in check, when required by the elite.
When you combine this with government, you have a recipe for "your life is over" workplace weather conditions. Maybe that storm only blows in once every 10 years, and doesn't take many people... so if you're high-functioning, you might not want to join the island where the heroes are fed to the Minotaur when the political climate goes South, sometimes without warning.
You might want to join the place where the worst-case scenario fail condition is a pink slip and a bad blog post. Naturally.
I think in an ideal world, questions about past use would never be asked and drug tests would be strictly limited to determining if someone is high/drunk on the job. The fundamental problem is employers losing potentially valuable employees due to their unnecessary and invasive drug policies. The polygraph isn't to blame.