You'd probably be wrong. Cultural fit may be important, and you don't want to get strange looks on an interview. At the same time the company still might be quite good - as in "early Google"-like good.
I've hold for a while the attitude that if a company pays attention if I'm wearing a suit in a job interview, I shouldn't get a job there.
I don't have anything to complain about that attitude. In retrospect, it saved me from quite a few bad situations that my inexperience would make me accept. But things changed and it does not apply anymore (am not even technically working on IT anymore).
Why? Are you saying that some company will give you a hard time because you are wearing a suit? That would certainly be humorous.
Humorous as in "Haha, I have no intention to work in such a place, ever".