You're conflating the value and price of an asset. E.g. if you own an asset like a stock that pays dividends, the value of that stock is the net present value of the future dividends. The price is determined by buyers and sellers and can diverge widely from the value.
It's related to the distinction between investing and speculating. Investing is based on whether the intrinsic value is below or above the price, while speculating is just based on the price.
Let me rephrase that: it takes a lesser percentage of bitcoin holders vs fiat currency holders selling their assets to collapse the value of their respective currencies.
Or even better: it's nearly impossible to do that with fiat currency, because there are checks and balances in place.
You can say the same about literally any asset or currency. If everyone sells, and no one is buying, then yes, the value drops to 0.