For what it's worth, as a white male, I've never had a bad experience opting out. At the very worst, they're curt and overly professional. Most of the time they make small chat.
I have had bad experiences twice. I am polite but refuse to answer questions or make small talk. Twice the man moved his hand "upwards until resistance" very fast and into my testicle. I complained and the managers response was "he has to go up until he feels resistance." TSA Complaint made, response was "no substantiating evidence." Shades of a police state.