It may be that MS-DOS/Windows has been "open enough" to capture and hold overwhelming market share. MS-DOS became the default PC OS in the 1980s due to Microsoft's cheap, non-exclusive licencing model, and Windows started out as a GUI layer on top of DOS.
In addition, third party developers have always enjoyed free, open access to the same set of system APIs as Microsoft itself, so it has been an easy platform to develop for. That, in turn, has strengthened the positive network externality that keeps Windows dominant.
The main reason Linux is not prevalent on the desktop has nothing to do with openness--it is just basically impossible to buy a new computer with Linux.
The reason it is impossible to buy a computer with Linux is largely Mircorosfts wildly anti-competitive behavior in the past. Note how much work and marketing it took Apple to carve out a non-trivial market share--there are no companies pushing Linux adoption nearly as heavily.
There are too many confounding factors affecting Linux on the desktop to use it to judge open technology in general.
That may be true but the assertion was that open systems tend to win. If you add the premise "except when there are external factors", there is really no argument left.
You're cherry-picking an example. Linux in general is a counter-counter example. In addition to running a large proportion of webservers (actually the open-source Apache server hosts over 50% of all websites), Linux also runs on the majority of smartphones sold.
Another way of looking at it though is that businesses choose open technology platforms, but consumers choose integrated/closed end products.
I'd like to believe that consumers will, in the end, choose open technology (directly or indirectly). Maybe it has something to do with the rate of technological change, and it's just a matter of time before the benefits of open technology are demanded. But the evidence is not conclusive.
If anything tips the scales, it will be when consumers start to figure out that they don't own their devices or the content on them. People are getting a vague sense of that with locked-down mp3 players that make it hard to copy songs around. It will be a bigger deal when the person that gives all their books away because now they have electronic ones becomes furious when they find out they are forever beholden to amazon (and its book-deleting whims) to be able to read them.
Right before our eyes, consumers are choosing an open technology platform for smartphones - Android - which had 52.5% market share on smartphone sales in Q3 2011. Compare 16.9% for Symbian, 15% for iOS, 11% for BB and 1.5% for Windows Phone.
But fashion changes quickly, and that can work in favor of openness. Investors don't invest huge amounts of money for something that will only be in fashion for a short period of time. So the solution is to build on open platforms rather than an entirely new proprietary stack.
There are other factors, too. I'm just saying that it's not entirely one-sided.
Seems like you're equally guilty of cherry-picking examples. Nothing you said is a counter-example to Linux on the desktop, which has less market share than two "locked in" OSs.
In addition, third party developers have always enjoyed free, open access to the same set of system APIs as Microsoft itself, so it has been an easy platform to develop for. That, in turn, has strengthened the positive network externality that keeps Windows dominant.