[cebert]

I get nervous every New Year’s Eve due to date/time issues. I work on emergency 911 software. In our system each time a 911 Call is created, we create an incident number in the format YYYY-NNNNNNNNNNNN where N is an incrementing number. I was oncall a few years ago when a date time bug was introduced that resulting in numbers being created prematurely by a few hours. As each hour passed more customers in a different time zone called in to report the issue. I was the only person working and was getting hammered with cases.

It sounds like an easy isssue to correct, but downstream systems that consume those numbers had already processed them and associated reports and other records with the incidents. I spent the next few months sorting out that mess and helping work with partners to clear out data.