It's not just that nobody cares enough to struggle through with you, but people actively will try to speak English with you as a way to get more proficient in English.
Yes, people will want that. My stock answer was "thanks, but anyone who lives in <x> should get used to speaking <y>" and that always settled it. That problem is more of an excuse for failing to learn proper <y> than a reason.
(There are a lot of expats among the parents at my children's school and it's easy to tell who persisted and who didn't.)
I used to have this happen when I lived in Germany and was trying to learn German. It would often lead to a funny situation where I would speak German and the German person would speak English. Given that I'm pretty stubborn, usually they would eventually switch back to German...
(There are a lot of expats among the parents at my children's school and it's easy to tell who persisted and who didn't.)