Well, you put a lot of trust in the individuals in this case.
A disgruntled employee can just let the bad guys in on purpose, saying "Yes they belong here".
That works until they run into a second person. In a big corp where people don't recognize each other you can also let the bad guys in, and once they're in nobody thinks twice about it.