[sghael]

I agree. My other thought was 'isn't there a staging server in there somewhere?' Something that is near identical to production, with fake production data, etc, that could surface the problem before a customer sees it.

btw, props to Fog Creek and OP for airing their dirty laundry. They take some heat, but in the end we all learn from it.

[gecko]

We have more than staging servers: we have staging accounts. I documented our full release process at http://news.ycombinator.com/item?id=2301680.