[aaco]

"... but anyone ever leave the software world to do something unrelated?"

Yes, however I don't know how large the number of people who leave programming related jobs is. It would be interesting to know some numbers about this topic.

Anyway, I don't think these numbers really matter for your decision. What matters is how you feel about that. I think the best thing you can do is talking to people who were in a similar situation before deciding what you're going to do next.

Some time ago I posted a quite interesting story here on HN of a game programmer at Electronic Arts who left his job to be a coffee farmer:

http://www.konaearth.com/Life/2006/060430/

Worth taking a look.