Even if they verbalized their knowledge incorrectly they give you something, which if you chose, you could further test / replicate. In other words even if they're BSing they're still falsifiable, not so "magic models" when their publisher may not want them falsified
Some are better at it than others, and no doubt many are pretty bad at it, but I have yet to see a neural net explain to me, accurately or not, why it came to the decision it did.
Don't forget that we can listen to their attempt at verbalizing that knowledge and then, in turn, draw/verbalize our own sketchy conclusions....and so on.