If you can't make yourself understood to the average engineer at a company, I think it's fair to say there's not a good fit between you and the company.
I'd say that if you're in a technical interview with a non-technical manager, you may've chosen the wrong company to interview with. And if it's a non-technical portion of the interview and you have trouble making yourself understood (to a manager), you're back to the "it's not a good fit" conclusion.
It was definitively the "technical interview with a non-technical manager", and yes, I do agree with you, but it's difficult to judge companies beforehand.