They really got their ass handed to them by Sony, Fuji and Panasonic more than anything. The market for high quality interchangeable lens cameras hasn't gone anywhere and isn't going anywhere, but Sony and co really shook it up by modernizing it, and made a lot of Olympus tech obsolete. I'll never understand how Olympus decided to sell a camera with a sensor a fourth the size of an A7 at the same price.
That said, the market for point and shoots and other low end to medium end cameras got really eviscerated and Olympus couldn't find a way to compete at the higher end.
Oh boy, there is way more to photography than sensor size. Olympus was a pioner in many of that and not surpassed in some of that.
They have the best IBIS, some innovative computational photography stuff and amazing weather proofing.
I am glad Olympus decided to sell M43 cameras, as those are you actually take with you.
Of course there is more to photography than sensor size. But an Olympus E-MD I was more expensive than a Sony A7ii that also has very good IBIS, can adapt pretty much any lens, and also has solid computational photography abilities.
The Sigma E-MD I that actually has all of the features you're talking about was almost as big as an A7. The lenses were smaller, but if you don't mind horrible apertures that you have to get on M4/3 you can also have very small and light lenses. Even moreso if you go APS-C.
The lens system also has huge pricing issues. The f2.0 zoom on M4/3 is equivalent to the kit lens on a Sony A7ii. That's a ten times price reduction!
My A7 with a pancake lens is small enough that I can actually slip it into my pocket (!)
Compromising image quality for portability at such an extreme extent is not a good idea when smartphone are so plentiful. If sensor size and optics doesn't really matter, then why not take pictures on your phone? They have amazing IBIS, innovative computational photography, and so on.
That said, the market for point and shoots and other low end to medium end cameras got really eviscerated and Olympus couldn't find a way to compete at the higher end.