It doesn't adjust immediately though. If a large majority of miner suddenly disappears then the block mining rate does indeed drop until the next difficulty adjustment.
So if 50% of the mining power left the next block would take on average about 20 minutes. Where it routinely takes about that long, because that’s just the way statistics work. A non issue.
Here’s a graph of the time blocks took over the last three years:
First, 20 minutes in average for power-distributed time-to-block would result in much, much higher peaks. In your chart I see 25 minutes for a block, then it will be 50 minutes.
Second, instead of two weeks to hash rate adjustment, it will take four weeks.
And if these staying with this slow bitcoin would decide to leave to more profitable currencies (not necessarily Bitcoin, there are other SHA256-based PoW schemes), that will push hash rate adjustment even further into future.
These 25 minute peaks were caused by a similar event, witness the also slow rates in the surrounding blocks. So that’s approximately how bad it would get in this ‘disastrous’ event.
Here’s a graph of the time blocks took over the last three years:
https://bitinfocharts.com/comparison/bitcoin-confirmationtim...
Look at the peaks and try to remember the issues that resulted in.