I'm not a programmer at all, and I found this answer quite obvious. Where else does new binary software come from, if not from a computer running older software?
It shouldn't be such a mystery, because it follows a biological model -- small, simple organisms produce more complex descendants by way of natural selection, all fueled by proteins. The same process creates complex software systems out of simple predecessors, all fueled by coffee.
> The chicken or egg problem shows that at some point you lose the connection to the predecessor and the process goes around in endless circles.
No, this isn't so. It's a myth about biology that's repeated by people who don't understand natural selection. An egg is a chicken's way to make an insurance salesman.
You don't have a chicken/egg problem with either biological evolution or computers. With biological evolution abiogenesis is at the bottom of the turtle stack, with computers it is likely some guy poking holes in some cards or rewiring a computer by hand.
Ultimately you don't need life to create life, and ultimately you don't need a computer program to build a computer program. There are other methods, even though we don't see those methods executed today.
If you think through, it is not _that_ obvious; it can't be turtles all the way down. So, where did the first binary software come from?
The answer is that somebody punched in a program from a computer's console, probably somewhere in the early 1960s, but possibly earlier. That program was used to make it easier to enter other programs, etc.
Given that copying working code is so simple, chances are that most of the currently running hardware can be traced back to a few persons entering a boot loader by hand.
That might even be one person. For example, Windows NT was initially developed for Intel's i860 (http://en.wikipedia.org/wiki/Windows_NT#Development), so chances are that it was bootstrapped from a Unix machine. Because of that, it may trace back to the same initial boot loader as, for example, Mac OS X.
Well, to be detailed, both the first kernel and the first compiler was probably entered from a stack of punchcards or punch tape made with mechanical "typewriters"; punching a program from console is rather unwieldy and generally you should do that only for the bootloader that loads the rest of the (manually compiled) code from some electromechanical device.