And in my experience the opposite applies as well. Having a PhD in a field providing the necessary background in theory doesn't necessarily imply the skills and experience required to produce useful models.
That's a really good question. The best people I've seen so far have had a willingness to rigorously align their models with reality, plus the knowledge and/or experience to know what to check. Verifying assumptions, picking evaluation metrics appropriate for the problem, checking for model interpretability, checking that model decisions are sane, and so on.