[qurren]

You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality.

You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).

[mightyham]

Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager.

In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience.

COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.

[eventualcomp]

Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org.

Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very different attitudes towards COEs and leadership principles.

[dlenski]

This was very much my experience, having worked in two different sub-organizations at AWS, and on several different services, in two different countries.

There's just extreme variation in the quality of the management, the quality of the engineers, the operational/development role split, the on-call schedule, and the development and testing methodology.

[awakeasleep]

Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve.

If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations.

That entails a real loss of face for them, and because they are the ones who actually run the show, it would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.

[qurren]

From the manager's side, you're absolutely correct. The SEV looks bad on you, and is a headache to document.

From the perspective of people you manage, it's a very different picture.

We (I say "we", because I was an IC) sit under you, and every year at performance review time you're effectively required to put some percentage of us in the "LE" bucket. Never mind that we could theoretically all HV3+ if you measure by "normal" peoples' standards, your manager isn't going to let you mark all of us as HV3 at the performance meetings. I know this, because I've been there as well at those meetings where truly high performing people were downrated to fit a distribution.

So what happens? When I see a peer's critical lurking bug, I have no incentive to fix it for the sake of prevention. If I fix it quietly so that it never surfaces, it looks like I haven't done any work for the week, or have done un-impactful work, and I get the stick from you. Preventing fires doesn't look like work, to non-technical eyes.

The only "safe" way to play this game of "survivor" is to let that bug surface eventually, then when the SEV comes up, I jump in and fix it, earning your approval, skip approval, VP approval, as well as potentially the other person gets the stick from you, because you have to give the stick to someone anyway, you get a reason to stick it on them. At least it's not getting stuck on me.

I'm sorry if this comes off as shocking to you, but it really shouldn't; the incentive structure is NOT set up for teamwork, plain and simple. If "putting customers first" is a value, then it absolutely needs to start from systematic changes of how people are managed.

[smallerize]

> > everything is driven with sticks instead of carrots

> this is not the case

> [describes all sticks and no carrots]

[dlenski]

> COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening.

Absolutely not my experience at AWS.

All the teams I was on treated them as "not a big deal", kind of a non-punitive exercise in technical writing, and the COE was always assigned to be written by an engineer who was not involved in causing the COE.

Also, the kinds of issues that did or didn't lead to COEs appeared to be largely random. I was considered to be an extremely good operational trouble-shooter on the team where I spent most of my time at AWS, and I was never able to predict what an L7-8 manager would decide was COE-worthy.

[RandomThrow321]

Obviously experiences vary greatly, but I was on one of the largest AWS orgs and they were quite punitive. People would demand them for perceived slights, then assign reviewers that were tough or on their side. Many of my friends had different experiences, though.

[swiftcoder]

Some orgs also use them to extract concessions from dependencies further up the chain. Upstream service won't fix an issue that's been causing problems for us? "unintentionally" let it become a SEV, so that we can send the CoE up the chain and get Jassy to drop the hammer on that team...

[dlenski]

Hmm. I'm curious about which org that was.

I spent the slight majority of my time at AWS in RDS.

[embedding-shape]

> Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again

And this surely works out great and completely prevents any issue, and this "inaccurate estimated billing data" issue this submission is about is both the first and last time this issue ever happens at Amazon?

[nullorempty]

This ^^^ amplified by indifference and not giving a shit caused by "AI Adoption".

There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job.

And no one is shooting for better than OK.

[hoppp]

Yeah but this also negates the argument that people gotta use it or they get left behind.

If it takes no skill or intelligence then nobody can get left behind because it's very fast to get back in.

[tyre]

Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.

[mendigou]

I am former AWS and this is pretty accurate.

The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).

[akdev1l]

Rube Goldberg machine attached with used bubble gum and somehow the bubblegum was chewed in all the wrong ways

[NBJack]

And many of which have been shipped across the pond to be someone else's problem.

[zelphirkalt]

Maybe they are former AWS employees for a reason and now want things to go better than they were at AWS.

[tmpz22]

AWS has had this reputation for over a decade. Every former AWS (including poached not fired) has relayed to me a verson of this.

Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect.

[embedding-shape]

> Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect.

The strange thing is that I only come across those missionaries online and never in person (although I did go to a AWS event once [never again] and met a bunch of them, so seemingly they are actually real, to my surprise).

[qurren]

Yes, I am a former AWS employee.

I got put on Focus because my "contributions were not coming through" to leadership.

[geodel]

"Former" seems to an important detail here.

[switchbak]

If I worked at a place like that, I'd sure as hell work my butt of to get a job somewhere else.

Or in my case, actively ignore any and all recruiting from that sesspool.

[gleenn]

If someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.

[FabCH]

Not the OP, but:

The point was not that their opinion is suspect, the point was that they are former because people who care about the customer get fired and/or that everyone who cared is former, so nobody who is left cares.

[thewebguyd]

> take the 3am on-call and be the hero instead

Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?"

I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.

[qurren]

The trick to surviving under such management is to jump in and put out other peoples' fires but not spend time preventing them even if you know how to.

[rootsudo]

This, exactly 100%

[dice]

How did you swing that transition? Did you study for PMP before applying around, leverage network to get in the door then backfill skills, or what?

[thewebguyd]

Yeah, I got my PMP before applying around, combined with some luck I suppose. My IT role was basically a solo sysadmin before where I basically was the technical PM + engineer in one, and I did that for about 8 years so I had a ton of experience I could spin on my resume.