Yeah, me either. They are definitely not designed to be worn over glasses comfortably. I ended up getting custom lenses that clip over the headset lenses because I play Beat Saber enough to justify it.
As a fellow glasses-wearer I can say that I've had a good experience with the Valve Index, which lets me adjust the distance between the lenses and my face enough that my glasses fit comfortably.
I've read that it's also possible, at least for some headsets and levels of correction, to purchase custom lenses such that you don't need glasses at all while wearing the headset.
I have a Quest 2, and I 3D printed some inserts from Thingiverse. They were designed to be used with the lenses from a specific set of glasses on Zenni Optical, which I ordered in my prescription (~$20).