I definitely would not want to have my phone as a key. I can keep my car key safely attached inside by backpack but I use my phone all the time when walking (taking pictures, OS maps, audiobooks) so its much more likely that I would damage/lose my phone than lose my car key.
I hope you have phones worth of $14k in your pocket in case you break one. I always keep my Tesla's "CC key" with me as a backup for the phone. Also, the phone is horrible key when you have stuff in the hands or using gloves in the winter.