I'd notice there was a problem when I stopped receiving mail and my current card stopped working. Even then, even if they DID get the new card, I could report any subsequent transactions as fraudulent (honestly, my mail being redirected would be a much bigger issue to me than someone having access to my card, so that doesn't add much to the attack potential).
Also, at some point, it becomes infeasible enough (that someone would have redirected my mail, hijacked my phone number or managed to change it with the bank, triggered a call from my bank, and managed to line them all up so I hadn't noticed there was a problem) and more trouble than it's worth to be worried about it happening.
Also, at some point, it becomes infeasible enough (that someone would have redirected my mail, hijacked my phone number or managed to change it with the bank, triggered a call from my bank, and managed to line them all up so I hadn't noticed there was a problem) and more trouble than it's worth to be worried about it happening.