On the other hand, it's getting more and more expensive. The amount of cheap oil still on the ground isn't nearly as large as the amount of oil.
Anyway, the main concern for climate change is coal. Not only it pollutes more, but there is much more of it on the ground than oil and gas. Luckily, solar is successfully competing with it.
Price is not a clear signal as you seem to be expecting. Remember, it's only a scalar that has to satisfy all of the constraints every person thinks is important.
Oil in particular seems to be unable to stay at very high prices. Historically we just get a recession and it goes down again.
Anyway, the main concern for climate change is coal. Not only it pollutes more, but there is much more of it on the ground than oil and gas. Luckily, solar is successfully competing with it.