Maybe they just want to facilitate you by letting you voice your contact and send message or call your friends by voice. I don't have a car with Android Auto, can you deny the permission?
I use LineageOS which has something called PrivacyGuard. I can give the app permission to access my contacts but then deny it at the PrivacyGuard level. I'm pretty sure that it just returns an empty contact list in this case.