[throwaway848483]

I think you don't need to invoke the heat exchange with the bottle to be relevant. I also believe that you get to accumulate it but there is a negative feedback due to the temperature gradient which prevent you to reach a high temperature difference.

For what it's worth (I'm not a physicist), here is my mental model of how I think it works. We are working at equilibrium we don't care about the macroscopic flow of air as it balances out. When the wind blows in the bottle it creates a more ordered arrangement of the air molecules (imagine laminar or vortex) at the neck (zone of low entropy). The internal energy will diffuse toward the region of more order (Second principle). So internal energy of the inside air flows towards the interface where it then diffuse into the nearby flow and get carried away by the wind.

As soon as there is a temperature difference between the inside air and outside air, traditional heat exchanges also occurs at the interface which has an opposite effect.