Would "I don't think your company is a good fit for me" be better wording (to show it's not your skills but their culture), or would that be too aggressive?
And perfectly fitting in such a situation. Being polite and professiobal avout breaking an interview of doesn't mean you cannot be "arrogant" doing it.