A company with Amazons resources could keep these fringe services around as long as there are paying customers. Just increase the price of the service to match the costs of keeping the legacy service around.
Three problems with this approach.
1. The ongoing “ktlo” (keep the lights on) maintenance cost for a minimal public aws service is on the order of 6 SDEs. Theres a constant stream of security and integration improvements like new IAM policy features, KMS handling, billing & metering, console chrome, docs, etc. These are generally nonnegotiable to keep _any_ sort of consistency across AWS.
This is discounting most operations/operational support, as you cant practically scale on oncall rota that small and your example is probably referring to a small customer base and a very infrastructure light service. You _could_ move that to a cheaper dev center & more shared ops, but that in itself is a lot of work.
2. Theres a lot of overlap between these services. Its often not complete, and theyll have different approaches. But fundamentally Amazons distributed nature means multiple orgs will solve similar problems that are adjacent but outside their focus, and then realize that “product” is someones elses better owned feature. Culling needs tk happen at that user experience level sooner or later. The linked blog post is a good example.
3. “Pay the average cost” is going to be shocking to see how much some of these cost to run. This cost increase is going to feel like extortion for customers who _already use other service_ and probably pay a lot more. Earning ongoing negative feedback over a minor service is not worth it. Nearly all services are priced based on a multi year forward looking marginal cost model. I would very much expect the average cost to be 10x the customer price for a small service like app mesh that never “reached scale” or got their P&L together. Then add in those dev salaries I mentioned.
Most importantly those ktlo dev hours in point #1 are an opportunity cost for AWS. They should be spent on new revenue generating work, not cost reduction, and certajnly not a dead end. More than 10 years ago the rough rule of thumb was that a dev year of work should be targeting $12M/yr in revenue generation. I know we didnt consistently hit that, but thats the kind of benchmark that would be used to evaluate staffing. In AWS today if the overall product doesnt have a line of sight to $500M its not a viable innestment.
It definitely scales of complexity/surface area, and Id say that straight number is a lower bounds fixed cost that you might see from an unsuccessful “small” service. In larger orgs its better a percentage, because its relative to overall investment/growth. My personal goal would be 20% ktlo, usually much closer to 40% in practice. All based on my personal experience doing the cloud thing across 2 companies for 15 years or so.
And I will give credit to AMZN/AWS, of old at least, that the above would be common knowledge with an expectation of much more org specific details for anyone sr manager/PE through VP whod be involved in roadmaps, HC allocation, etc.
are we assuming only 9-5? 2.3 SDE could do that, but i don't get 6. a 24/7 operation needs 2FTE and 1 PTE per shift, and there are three shifts.
And i suppose "6 SDEs" for a service that has... what number of MAU or whatever?
me and a great friend of mine split our roles with our "projects", he deals with paid clients, and i deal with our required services (email, matrix, nextcloud, pastebins, misskey, PBXen, wireguard/VPN/Proxying...) and there's just two of us. I can't even think of the last time we had downtime, except at the tail-end of last year, one or the other of us caused a fault in proxmox, where commands were no longer issuing, and we had to do a reboot and fsck, the drill.
I understand at "AWS" scale obviously there's going to be more fires than 1 or two people can put out, but 6 SDEs is over 2 million in compensation all told for amazon, right?
So i guess what i am asking for is context. I've run stuff with hundreds of millions of monthly unique (for a paycheck). That company had over a billion uniques across all of their verticals 14 years ago. One network admin (like senior, but still). When i was there, there were 4 DBAs. there was 1 who could deal with developers, the oracle/postgresql guy who is one of my favorite humans ever - he could convert C-level grunts into sql queries or whatever it is they do. Then there was a part time mariadb; and then a full time Oracle DBA for just 1 site. I'm really straining to remember who else was basically infra/ops and it was less than 6[^1] of us for the whole company. We did have a NOC in another country but prior to my arrival all they were really trained to know how to do was page the correct person in the US. Before i left, 3 of the NOC people moved to the US and started working at the office i did.
If you know anyone else that's worked for this company, you'd know. If you haven't used one of their sites today, you probably will this week.
sibling and other comments note that amazon has "consistent look and feel and management and..." so maybe all that "busywork" takes 6 full times?
[^1] i don't count the DBA team as KTLO since i can only think of a single instance when i had to be on a conference and they also were. hurricane in Va!
> 2.3 SDE could do that, but i don't get 6. a 24/7 operation needs 2FTE and 1 PTE per shift, and there are three shifts.
having the team constantly on call is a recipe for having them quit. sure, you could run this with the bare minimum number of engineers, but your turnover would be so high and given how high hiring costs tend to be, this is a net negative
> so maybe all that "busywork" takes 6 full times?
ktlo in a constantly changing company is not easy. software / host patching to maintain compliance is necessary busywork that requires a bit of babysitting. not to mention keeping up with required migrations to new stuff due to internal deprecations.
You’re forgetting one important point. No one who has any interest in their career inside or outside of Amazon wants to be on a service team for a dead service. That would look horrible on both their promo doc internally and when trying to get through a behavioral interview externally.
From all what I've heard over the years, they have a problem staffing the service teams. I imagine with the AI craze that just got worse, as they certainly shifted resources to work on that.
If they really have staffing problems, it'd make sense in the short-term to shut down services which don't bring in much revenue. In the long-term that might fire back if AWS customers loose trust in the services they're using still being there in the future.
The obvious solution to a possible staffing problem would of course be not fire a certain percentage of your employees every year.
I have a lot of grips with AWS, but long term support isn’t one of them.
They’ve given nearly 2 years notice here (more than double what a lot of other SaaS providers offer) and there will be an army of AWS engineers available to help customers migrate too.
AWS do actually discontinue services all the time. It just doesn’t make the headlines because their depreciation process is so good.
Now, if you wanted to discuss their billing, difficultly to navigating the literally thousands of products they offer, contradicting documentation, half baked implementations, cryptic error messages (assuming you’re even lucky enough to get an error) when deployments fail, or the dozen other hurdles that make using AWS a highly paid specialty, then I’d agree with you. But depreciation of services is one of their strengths.
> In the long-term that might fire back if AWS customers loose trust in the services they're using still being there in the future.
Where are they going to go to get higher guarantees of services not being abandoned? Google?
> The obvious solution to a possible staffing problem would of course be not fire a certain percentage of your employees every year.
Have you worked for a BigTech company and seen their promo doc? No developer would want to work on a dead end service who was interested in their career.
Really? I've heard Amazon is so understanding of the difficulty in new hires picking up legacy code, even with the extremely diligent help of the other "mates" on their "team".
I'm sure with the excellent work life balance, supportive atmosphere, nurturing organization, and upbeat attitude, Amazon's staffing issues are just a short blip.
One thing Amazon is known for us a good place for a long term stable career!
It's a good thing that organizations employment is so stable with so many foundational services that so much of our economy depends upon.
For them it's 0.00...1% of their business. For a small customer who deployed on their service it might be 30% or more of their development budget. Of course that's the way corporations do business. But that's why as a small one you can't really trust any of them. You can just place bets and sometimes you lose.
To put it another way, it is an implicit cost of using AWS: You have to have contingency plans to cope with discontinued services, and you need to be able to realistically execute those plans. I think it is just another reason that AWS isn't as good a deal as it might initially appear to be for small businesses. (Larger organizations may have a better chance of amortizing these costs over many projects.)
This is discounting most operations/operational support, as you cant practically scale on oncall rota that small and your example is probably referring to a small customer base and a very infrastructure light service. You _could_ move that to a cheaper dev center & more shared ops, but that in itself is a lot of work.
2. Theres a lot of overlap between these services. Its often not complete, and theyll have different approaches. But fundamentally Amazons distributed nature means multiple orgs will solve similar problems that are adjacent but outside their focus, and then realize that “product” is someones elses better owned feature. Culling needs tk happen at that user experience level sooner or later. The linked blog post is a good example.
3. “Pay the average cost” is going to be shocking to see how much some of these cost to run. This cost increase is going to feel like extortion for customers who _already use other service_ and probably pay a lot more. Earning ongoing negative feedback over a minor service is not worth it. Nearly all services are priced based on a multi year forward looking marginal cost model. I would very much expect the average cost to be 10x the customer price for a small service like app mesh that never “reached scale” or got their P&L together. Then add in those dev salaries I mentioned.
Most importantly those ktlo dev hours in point #1 are an opportunity cost for AWS. They should be spent on new revenue generating work, not cost reduction, and certajnly not a dead end. More than 10 years ago the rough rule of thumb was that a dev year of work should be targeting $12M/yr in revenue generation. I know we didnt consistently hit that, but thats the kind of benchmark that would be used to evaluate staffing. In AWS today if the overall product doesnt have a line of sight to $500M its not a viable innestment.