You're basically never 100% safe. In this particular case, the broker at which you bought the index fund units can scam you. It actually happened in Poland maybe 10 years ago - managers at a respectable brokerage firm were taking in orders from people but, instead of buying the requested papers, bought something else which they believed would be more profitable. They ended up being wrong and, when the clients wanted to eventually "sell" their papers (which were never bought in the first place), the company folded as it didn't have enough to pay the clients.
If memory serves, The largest single-day drop was around 25% and the biggest total was around 90%. That leaves 10% to live off of: if you have 10x what’s necessary for the rest of your life invested you can absorb those losses.
To have 10x what's necessary, you need to be able to survive on 1% returns (I'm trying to be generous there). So if you can live on $30,000, you need to have "only" $3MM invested.
You should also not be investing more than 25% of your assets into a single asset class, so you need to have $6-12MM of assets (range depending on considerations) in the first place.
I think having even $3MM is a reasonably called 'rich', but others may disagree.
I don't think the people who are non-obvious rich put their money solely in an index fund. They'll generally have about 50-500 apartment units in some big city, which (if properly insured) guarantees you can weather anything except for wars and asteroids. Maybe they put their earnings into an index fund though ...