[bryanrasmussen]

reading this I keep thinking the real questions are:

1. probably the notice sending and db deletion is two separate teams or responsibilities. Are they?

2. Did people know there was a bug in notice sending so some notices were not being sent, all notices not being sent? I ask this because generally in places I've worked where notice sending was an important part of things you knew if there was a bug and notices were not being sent. But maybe it wasn't that important for Heroku. Maybe it was not known that notices were not being sent for a while - or was it known immediately but things on other parts of business chugged along anyway.

3. If they knew notices were not being sent and they went ahead and deleted db anyway, seems messed up, but that would probably be ok with people if they had data retention for people who did not get notices sent.

4. The whole thing your stuff can be deleted at any time without telling you is basically probably true almost everywhere in that notice sending can have a bug and deletion of stuff is probably not adequately tied to notice sending so that if notice not sent automatic deletion is stopped. Which I'm thinking is probably everywhere - if you work somewhere with automatic deletion and a notice sending module - what happens? Is this scenario handled?

5. answer to this is probably not, but is there a legal issue if notices not sent and stuff deleted, issue might be if some notices were sent - if account A gets notice about deletion and is thus able to act on it and account B does not get notice and is thus not able to act on it there might be a ground for action. Probably not, but when something seems unfair there might be a law that can be stretched to fit it.

[mst]

My guess would be that a tiny percentage of accounts didn't get their notices sent, nobody complained about it because they didn't realise they were supposed to be getting them (or at least for whatever reason no such complaints were escalated to the people who would go "oh shit" upon seeing one) and some unfortunate combination of technical factors meant whatever internal monitoring existed didn't pick up the omissions.

Note that this is not to say the end result wasn't an indefensible disaster, only that disasters seldom have only a single cause and the above is my best uneducated guess at how things came together to cause this one.