[mightyham]

Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager.

In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience.

COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.

[eventualcomp]

Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org.

Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very different attitudes towards COEs and leadership principles.

[dlenski]

This was very much my experience, having worked in two different sub-organizations at AWS, and on several different services, in two different countries.

There's just extreme variation in the quality of the management, the quality of the engineers, the operational/development role split, the on-call schedule, and the development and testing methodology.

[awakeasleep]

Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve.

If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations.

That entails a real loss of face for them, and because they are the ones who actually run the show, it would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.

[qurren]

From the manager's side, you're absolutely correct. The SEV looks bad on you, and is a headache to document.

From the perspective of people you manage, it's a very different picture.

We (I say "we", because I was an IC) sit under you, and every year at performance review time you're effectively required to put some percentage of us in the "LE" bucket. Never mind that we could theoretically all HV3+ if you measure by "normal" peoples' standards, your manager isn't going to let you mark all of us as HV3 at the performance meetings. I know this, because I've been there as well at those meetings where truly high performing people were downrated to fit a distribution.

So what happens? When I see a peer's critical lurking bug, I have no incentive to fix it for the sake of prevention. If I fix it quietly so that it never surfaces, it looks like I haven't done any work for the week, or have done un-impactful work, and I get the stick from you. Preventing fires doesn't look like work, to non-technical eyes.

The only "safe" way to play this game of "survivor" is to let that bug surface eventually, then when the SEV comes up, I jump in and fix it, earning your approval, skip approval, VP approval, as well as potentially the other person gets the stick from you, because you have to give the stick to someone anyway, you get a reason to stick it on them. At least it's not getting stuck on me.

I'm sorry if this comes off as shocking to you, but it really shouldn't; the incentive structure is NOT set up for teamwork, plain and simple. If "putting customers first" is a value, then it absolutely needs to start from systematic changes of how people are managed.

[smallerize]

> > everything is driven with sticks instead of carrots

> this is not the case

> [describes all sticks and no carrots]

[dlenski]

> COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening.

Absolutely not my experience at AWS.

All the teams I was on treated them as "not a big deal", kind of a non-punitive exercise in technical writing, and the COE was always assigned to be written by an engineer who was not involved in causing the COE.

Also, the kinds of issues that did or didn't lead to COEs appeared to be largely random. I was considered to be an extremely good operational trouble-shooter on the team where I spent most of my time at AWS, and I was never able to predict what an L7-8 manager would decide was COE-worthy.

[RandomThrow321]

Obviously experiences vary greatly, but I was on one of the largest AWS orgs and they were quite punitive. People would demand them for perceived slights, then assign reviewers that were tough or on their side. Many of my friends had different experiences, though.

[swiftcoder]

Some orgs also use them to extract concessions from dependencies further up the chain. Upstream service won't fix an issue that's been causing problems for us? "unintentionally" let it become a SEV, so that we can send the CoE up the chain and get Jassy to drop the hammer on that team...

[dlenski]

Hmm. I'm curious about which org that was.

I spent the slight majority of my time at AWS in RDS.

[embedding-shape]

> Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again

And this surely works out great and completely prevents any issue, and this "inaccurate estimated billing data" issue this submission is about is both the first and last time this issue ever happens at Amazon?