Ice ages are, like the name say, ages. No problem if the Earth heating would take ages. We would have time to evolve 60° body temperature or something...
Also, when too cold, kill a sheep, take it's coat. When too hot, what are we (and our animals, food and pets) going to do? Undress our own skin?
Imagine humans during previous ice ages, no space heaters and The North Face jackets still having babies. We're so coddled in comparison.