I think a good idea might be to have TOFU and self-signed only as a fallback. If there was no initial mismatch, and then upate cert periodically.