|
|
|
|
|
|
by Johnny555
1241 days ago
|
|
|
I used to work for a small startup, and postmortems were truly no blame - engineers would talk about exactly what happened and wouldn't hesitate to put the blame on their mistakes. But as the company grew, the postmortems became more about blame since now you're not blaming an engineer, but an entire team so singling them out isn't personal. The postmortems were no longer a single engineer describing what happened in his code, but were team leads talking on behalf of teams. They were all about shifting blame from your own team and talking about why a service from another team led to the problem, even if your team could have (and should have) been able to work around it without melting down. I'm no longer at the company, but Postmortems are much more useful when they really are no-blame because you can get to the real root of the problem, but I don't know if that's possible in a large company. |
|
|
A few years back, a task to modify an index was given to a scrum team. The lead was away and the senior people could not be bothered. The junior developer stack overflowed an answer, asked for review, tested the script and let it rip. She missed that the change deleted everything if you noticed. Every environment, every data center wiped out. 10B records in each prod instance. Lessons were learned and processes fixed. She was not fired, but rather became one of the people safeguarding the keys to our prod kingdom as we fixed out broken process. I stole her away as my first report when I switched groups.