Hacker News new | ask | show | jobs
by fifticon 1072 days ago
I was in a successful company that nearly died in 2017, where our entire production system corrupted itself due to a sneaking scale bug we had ported the system into. The problem was, the system with data had been running migrated for 3 months with the bug in it, so it was no longer possible to revert to the earlier working design. We were down for a week where no clients could run, and we spent the next 12 months purely digging ourselves out of that hole, with all new development paused, and all hands on limping the ship along. I would say, that bug was very close to ending us. Luckily, we never disappointed in a similar way since.
1 comments

What I’m hearing here is that even a software problem so bad it forced you to focus on nothing except trying to fix it, for a full year, wasn’t bad enough to make the company go under.