The issue comes if someone can get you to accept their CA. In both this case and for MITM attacks on TLS. At that point it's game over.