Oh yeah. Walking around a manufacturing plant. Brand-new hi-viz vest over a dress shirt, walk purposefully. "Must be management."
I wasn't management, I was just new on site and had come from a planning meeting, and I was pretty sure the next guy I needed to meet was in the belly of the plant in a location I only had the vaguest idea of, but I knew it was gonna be a long walk.
I ended up walking the length of the place twice (almost a mile) before I decided I'd had enough sightseeing (but what sightseeing it was!) and and actually asked someone, who made a wisecrack about how he helps so many visiting managers in this joint they should make him a manager himself! I decided against correcting him.
Over the coming weeks at that plant, I wore a T-shirt and older hi-viz if I wanted to blend in as a worker, or a button-down shirt and the crisp hi-viz if I wanted to wander. As long as I kept abreast of my assigned duties, nobody sweated the details, and it was better than any museum of science and industry I've ever paid to get into. Had a few more folks make gentle cracks about how I must be a manager of some other group over to see what this group does, I'd ask a few questions and be on my way.
Likewise for jobsites and urbex, I have a hard-hat covered in stickers, and one that's so pristine I keep it in a pillowcase. Perceived wear is an important component of The Look™.
It varies. The more you can look like "the help" the more likely people are to ignore you outright. And the hi-viz + clipboard is a pretty generic "you don't know me, but I'm the help" outfit.
A few years ago I was once briefly directed into the cockpit of a plane to check on its maintenance status while I was boarding a normal coach flight. I was wearing my only clean jacket, a waterproof high viz coat that must be popular with people who work on rainy roads.
Depends: in many places the high-vis hits the “I'm wearing this because the rules say I have to and I'm not important enough to flout them even though they clearly don't matter here” note very well.
I don't think it would work in most situations. Construction site? Sure. Sporting event? Probably.
But in most of the places a person might try to go, I personally would avoid a high-vis vest at all costs.