Here's what the local CO looks like in my hometown (~6K people). It was still a SxS switch when I was in high school (early 90s) but has since been upgraded; I presume most of the building is empty at this point and it's just used as staging for repair parts (see the pic of the back fenced-in area).
Centrally located, no public entrances, and no windows. I imagine the lack of windows also helps make it more secure, and less vulnerable to vandalism.
And it's not surprising you'd never see anyone come or go. These offices typically don't have permanent staff; technicians stop by for maintenance as needed.
http://imgur.com/a/MMSK4