The problem is the millions to billions of people who are living in areas that will become virtually uninhabitable due to temperature changes and sea level rise. The first way climate change will seriously negatively impact humanity is through geopolitical conflict and a massive refugee crisis.
Change causes extinctions. If the temperature suddenly swings down, things die. If it subsequently swings back up, do they come back? No! More things die.