There's a big difference between "X would like to access your contacts" and "X would like to access your contacts, then store everything on their servers for reasons/duration of their determination"
But not from the OS's point of view. iOS (and Android) can only warn about the contact crossing the threshold into the app. After that the app can do as it pleases...