|
"Tell us a story about how you took down production." We had a real need for batch queries, so I built a declarative batch query mechanism for our in-house ORM. To do this, we needed to make actual objects representing database relations. Unfortunately, the "collections of lambdas" we were using as an ORM had started getting polluted with buisness logic. So I used a parser/rewrite engine to match the un-polluted DB transformation lambdas and to substitute DB column and relation objects implementing the same logic in the un-polluted lambdas. I checked in a massive change affecting 2500 of the ~5000 lambdas, syntactically matching the logic in all 2500 of those with the parser framework, and then also vetted everything by hand. I was 100% sure the logic was duplicated exactly. This would allow us to introduce the objects, and it would also stop the march of business logic into the DB framework. Everything checked out in QA, so we moved it to production. Production immediately came to a screeching halt. However, in under an hour, I discovered that a coworker had polluted an even lower level of the ORM framework with business logic. (The code that called the lambdas in the first place.) We replicated that logic in the new framework, and everything worked 100%. I also came out a huge hero in the end, because the batch query mechanism I introduced reduced the average number of queries for opening a portfolio from ~2500 to only 50, and because it was declarative, it could be reused by any programmer very easily. |
"I see. So you failed to communicate effectively with a coworker, which resulted in an hour of downtime. I didn't hear you mention anything about seeing this as a problem, let alone addressing it. Thanks for your time."