But then third party apps can force users to accept this before they work (here I am especially thinking of school and work apps that people might be forced to use).
App store guidelines already strictly prohibit developers from demanding overbroad or unnecessary permissions, and they expressly forbid locking unrelated features behind irrelevant data requests. Any app that attempts to do so can easily be yanked off the app store by a single complaint.