Some of the complaints I hear are difficulty breathing, and fogging up glasses. Seems to me that in such cases a different mask design might suffice. Some also believe that the masks aren't useful, so don't bother.
I suffer from both problems (fogging up glasses and difficult to breath) but I still prefer to wear a mask. It just means I go less to places where its mandatory to wear one. Which means I go out less, which is statistically good to combat the pandemic. My mother does the very same, except she's a risk group while I'm not. As does my partner. n=3 but I doubt we're alone. Even in a seemingly unlikely way such as this indirect example, masks work.