Avoiding all of the analogies... I'm on the technical side, and there's non-trivial work involved in evaluating the quality of the work of other developers.
Particularly for sniffing out security problems... you're fighting a many-headed beast, so unless you're quite technical yourself (and know how to exploit the whole OWASP list, for example), I can't imagine how you'd evaluate someone else's statement that "yes, this will be secure"; even if the lead developer can talk in depth about 10 common security holes, what if s/he simply isn't familiar with #11 and #12? Or lacks the creativity to notice how an architecture choice will severely hamper security in the future?
There's non-trivial work involved in evaluating my own work, and every now & again when I step back to view my own approach to a technical problem I change course.
So -- it's certainly possible to evaluate developers without being technical, but you're still forced to trust their diligence and skill quite a bit. "Talking the talk" of doing things right technically isn't very hard (just read a lot of dev blogs... you'll pick it up); actually doing them right with some consistency is a different beast entirely, and not everyone succeeds even with the best of intentions (...but this is much harder to evaluate).
Quality of software is pretty much irrelevant for startups. The code is usually crap and gets thrown away over and over anyway. The real focus is on people.
If the code is crap, that might explain why it gets thrown away over and over. In my experience, if code is well-written, it should pretty much always be possible to update, revamp, pivot, etc. Rewriting from scratch (especially repeatedly!) is a waste of time that a startup can't afford.
Engineers for startups are paid poorly and almost always work on technically uninteresting problems. Where are you going to get people who can write good code? And again, why would you bother if the hard part of startups is everything except getting the code to work?