Maybe rocks in the scree cool down the air in the pores between them. Cool air is denser so it sinks and flows out at the bottom of the porous layer, and new warm air is pulled into the ground towards the top of the scree pile. People found the places that cool air was flowing out and built these grottos there. Not saying I knew anything about these before 10 minutes ago, but that seems plausible to me.
IANAGeologist, but my understanding is that when a steady wind blows from underground it's often due to a naturally occurring trompe: water flowing underground entrains air bubbles which collect in caverns and leak back to the atmosphere. Trompes can be quite large.