It's mainly the lack of emergency services support in my country. Every time I called the operator can't see where I am through GPS, first question asked was what state im in.
I used GrapheneOS for about half a year as my primary phone OS. It does not scramble your GPS in any way (it has the same course/fine-grained GPS permissions as regular Android), but it does allow you to block a lot more app permissions. It's more likely that they haven't set the correct permission(s) for that information to bubble through to emergency services.
I would also be surprised if there weren't cell phone system-based fallbacks for emergency services. The carriers have a good idea of where you're at based on the towers you're connected to. There are plenty of situations where GPS doesn't work.