You'd be surprised. Ancient Iran (a literal desert) kept buildings cool with windcatchers [1] and qanats [2]. I do agree that we have a tendency to set up shop in really extreme places, but that's not necessarily a limiting factor in sustainable design.
Just keep in mind that deserts are the easiest places for artificial temperature control. Anything you try in a desert works, there are no clouds obstructing the Sun, or humidity stopping a passive cooler; every night is cold, every day is hot; there's nothing blocking winds or passive heat emission.
Basically, everything that makes them so damn inhospitable also makes it easy to manage the temperature of a building.
You will have a much harder time on any other place.
There should be some standard measure for how amazing a feat of historical civil engineering is and also how much we know how they actually did it, so 10/0 is 'aliens did it' through to 1/10 for basic mud huts.
Qanats seem like 'aliens did it' level of technological sophistication from my limited understanding of when and how they were built, but I guess that's more due to my lack of understanding than genuine alien influence.
But are there experts out there that know how they actually built them, and how would they rank qanats vs pyramids or great cathedrals? Like how many lifetimes were needed to plan and execute them?
Basically, everything that makes them so damn inhospitable also makes it easy to manage the temperature of a building.
You will have a much harder time on any other place.