This is where you and I are different. I’m neurodivergent so having another low-mental load activity (like cleaning) actually helps my mind from wondering.
If the has to do with being neurodivergent or neurotypical I have no idea, it might just be people being different, but low-mental load activities is what makes my mind wander off. It's also something I actively want to happen, hence working out without music, podcasts or audio-books. To me at least it's a kind of meditation and it calms my mind.