On mobile screens the cards that were previously selected take a lot of space in the background; if you click that background card, intending to “deselect” the foreground card, then you suddenly play the background card. But yes, if you click anywhere else then all cards are deselected. Still, I find this very unintuitive.
I changed the behavior now: tap and hold magnifies it and short tap select the card. At least with my fingers now I was able to avoid the mentioned side effects