Couldn't have said it better, learning from only listening can be really hard under several conditions when something happening in front of you can snatch your attention away and make you miss a big chunk of info.
That's what the back/rewind button is for, though for many purposes you'd be better off listening to lecture series than podcasts for learning. I know I learned a lot about development economics from listening to Duflo and Bannerjee's lectures.
Listening to podcasts for learning is not any worse than using wikipedia, considering wikipedia articles quality (not a popular reading, but a learning-grade one). Both are equally far from what's needed for learning.