[Anon1096]

The post is so dramatized and clearly written by someone with a grudge such that it really detracts from any point that is trying to be made, if there is any.

From another former Az eng now elsewhere still working on big systems, the post gets way way more boring when you realize that things like "Principle Group Manager" is just an M2 and Principal in general is L6 (maybe even L5) Google equivalent. Similarly Sev2 is hardly notable for anyone actually working on the foundational infra. There are certainly problems in Azure, but it's huge and rough edges are to be expected. It mostly marches on. IMO maturity is realizing this and working within the system to improve it rather than trying to lay out all the dirty laundry to an Internet audience that will undoubtedly lap it up and happily cry Microslop.

Last thing, the final part 6 comes off as really childish, risks to national security and sending letters to the board, really? Azure is still chugging along apparently despite everything being mentioned. People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true.

[axelriet]

>risks to national security and sending letters to the board, really?

Yes, really, and guess what the DoD did on Aug 29, 2025, exactly 234 days after I warned the CEO of potential risks?

https://www.propublica.org/article/microsoft-china-defense-d...

It wasn’t specifically about the escort sessions from any particular country, though, but about the list of underlying reasons why direct node access was necessary.

[jiggawatts]

> People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true.

Or… you’ve just normalised the deviation.

One of the few reliable barometers of an organisation (or their products) is the wtf/day exclaimed by new hires.

After about three or four weeks everyone adapts, learns what they can and can’t criticise without fallout, and settles into the mud to wallow with everyone else that has become accustomed to the filth.

As an Azure user I can tell you that it’s blindingly obvious even from the outside that the engineering quality is rock bottom. Throwing features over the fence as fast as possible to catch up to AWS was clearly the only priority for over a decade and has resulted in a giant ball of mud that now they can’t change because published APIs and offered products must continue to have support for years. Those rushed decisions have painted Azure into a corner.

You may puff your chest out, and even take legitimate pride in building the second largest public cloud in the world, but please don’t fool yourself that the quality of this edifice is anything other than rickety and falling apart at the seams.

Remind me: can I use IPv6 safely yet? Does it still break Postgres in other networks? Can azcopy actually move files yet, like every other bulk copy tool ever made by man? Can I upgrade a VM in-place to a new SKU without deleting and recreating it to work around your internal Hyper-V cluster API limitations? Premium SSDv2 disks for boot disks… when? Etc…

You may list excuses for these quality gaps, but these kinds of things just weren’t an issue anywhere else I’ve worked as far back as twenty years ago! Heck, I built a natively “all IPv6” VMware ESXi cluster over a decade ago!

[12_throw_away]

> One of the few reliable barometers of an organisation (or their products) is the wtf/day exclaimed by new hires.

Wellllll ... my observations after many cycles of this are:

- wtfs/day exclaimed by people interacting with *a new codebase* are not indicative of anything. People first encountering the internals of any reasonably interesting system will always be baffled. In this context "wtf" might just mean "learning something new".

- wtfs/day exclaimed by people learning about your *processes and workflows* are extremely important and should be taken extremely seriously. "wtf, did you know all your junior devs are sharing a single admin API token over email?" for example.

[zdragnar]

> One of the few reliable barometers of an organisation (or their products) is the wtf/day exclaimed by new hires.

Eh, I don't think this is exactly as reliable as you'd expect.

My previous job had a fairly straight forward code base but had fairly poor reliability for the few customers we had, and the WTF portions usually weren't the ones that caused downtime.

On the other hand, I'm currently working on a legacy system with daily WTFs from pretty much everyone, with a greater degree of complexity in a number of places, and yet we get fewer bug reports and at least an order of magnitude if not two more daily users.

With all of that said... I don't think I've used any of Microsoft's new software in years and thought to myself "this feels like it was well made."

[jiggawatts]

The rapid decay of WTF/day over time applies to both new employees and new customers.

> currently working on a legacy system

"Legacy" is the magic word here! Those customers are pissed, trust me, but they've long ago given up trying to do anything about it. That's why you don't hear about it. Not because there are no bugs, but because nobody can be bothered to submit bug reports after learning long ago that doing so is futile.

I once read a paper claiming that for every major software incident (crash, data loss, outage, etc...) between only one in a thousand to one in ten thousand will be formally reported up to an engineer capable of fixing the issue.

I refused to believe that metric until I started collecting crash reports (and other stats) automatically on a legacy system and discovered to my horror that it was crashing multiple times per user per day, and required on average a backup restore once a week or so per user due to data corruption! We got about one support call per 4,500 such incidents.

[zdragnar]

The customers aren't pissed, we're doing demos to new departments and lining up customizations and expansion as quickly as we can. We're growing faster than ever within our largest customer.

I also didn't say there are no bugs or complaints, I said the system is more stable. But yes, there are fewer bugs and complaints, especially on the critical features.

I didn't use the word legacy to mean abandoned, just that it's been around a long time, we're maintaining it while also building newer features in newer tech, as opposed to my previous company which was a green field startup.

[jiggawatts]

> But yes, there are fewer bugs and complaints

How do you know?

By that question I mean: Do you think there are fewer bugs because you hear fewer complaints from humans, or because you have a no-humans-involved mechanism for objectively evaluating the rate of bugs?

Even if you have a mechanical method for collecting bug reports, crash logs, or whatever, that can still obscure the true quality of the codebase.

One such example that I keep thinking about was the computer game Path of Exile. It has "super fans" that all have 10,000 hours of playtime that will swear up and down that it is one of the best games ever. When I first played it, I found so many little bugs and issues that I had more fun jotting them down than actually playing the game! I collected pages and pages of bullet points. None were crash bugs that would have been logged, and every one was the type of thing that players would eventually learn to work around by avoiding scenarios that caused the issue. I.e.: "Don't click to fast after going through a door because your orientation will be random on the other side, so you might be sent back to where you came from", that kind of thing.

Honestly and objectively measuring the quality of a software application (or any product) is hard.

[justincormack]

I mean, the org had already decreed everything needed to be rewritten in Rust according to the account.

[bawolff]

> Last thing, the final part 6 comes off as really childish, risks to national security and sending letters to the board, really?

That struck me too. Maybe i've never worked high enough in an org (im unclear how highly ranked the author of the piece is) but i've never been in an org where going over your boss's boss's boss's boss's head and writing a letter to the board was likely to go well.

That said, i could easily believe that both Azure is an absolute mess and that the author of the piece was fired because of how he went about things.

[axelriet]

I didn’t say it went well. Actually I said it didn’t go well :(

[bawolff]

Yes, but you are writing as if you expected it to go well.

[rhyperior]

I’d say they’re writing as if they expect everyone to share their ideals, and they’re responding to violations of those ideals in a naive and disempowered way. That doesn’t mean they’re wrong in those ideals, but the way they tried to fix things… they interacted with Cutler, why didn’t they try to influence him to fix things? Or any other senior technical leaders?

[axelriet]

I never said I didn’t. It was a multi-year escalation and I shared my concerns widely along with concrete options.

The thing is in corporate environments people avoid admitting anything is wrong because that would make them look bad and also disavows the bosses who promoted them, so the true best interest of the company takes a back seat.

[aswegs8]

Yeah I get that, I have been in a similar situation. At some point you just want to pull the trigger instead of going along with values that aren't yours. Sure, there is always the way of just silently disagree and move on, but who cares. We live one life not to maximize our careers, but to be who we are. Was an interesting read and wish all the best to you in the future.

[kraemahz]

AWS and Google Cloud are both huge and are significantly better in UX/DX. My only experience with Azure was that it barely worked, provided very little in the way of information about why it didn't. I only have negative impressions of Azure whereas at least GC and AWS I can say my experiences are mixed.

[com2kid]

> From another former Az eng now elsewhere still working on big systems, the post gets way way more boring when you realize that things like "Principle Group Manager" is just an M2 and Principal in general is L6 (maybe even L5) Google equivalent. Similarly Sev2 is hardly notable for anyone actually working on the foundational infra.

Before the days of title inflation across the industry, a a Principal at Microsoft was a rare thing. When I was there, the ratio was maybe 1 principal for every 30 developers. Principals were looked up to, had decades of experience, and knew their shit really well. They were the big guns you called in to fix things when the shit really hit the fan, or when no one else could figure out what was going on.

[Anon1096]

One of Microsoft's problems is their pay is significantly lower than FAANG and so you very very rarely see people with expertise in the same verticals jump to Azure. I get that "the deal" at Microsoft is lower pressure for lower pay but it really hinders the talent pipeline. There are some good home grown principals and seniors, but even then I think the people I worked with would have done well to jump around and get a stint at another cloud provider to see what it's like. Many of them started as new grads and their whole career was just at Azure.

Meanwhile when I was at another company we would get a weekly new hire post with very high pedigree from other FAANGs. And with that we got a lot of industry leading ideas by osmosis that you don't see Azure getting.

[fleshdaddy]

Yeah the deal has also changed. Right as I was leaving the messaging started changing a lot and there was a clear top down “you all need to work harder”. They hired an ex Amazon guy to run my org which really drove the message home.

To be fair though I think Microsoft has decided they are fine with rank and file being mediocre. I don’t know how interested they are in competing for top talent except for at the top.

[com2kid]

> I get that "the deal" at Microsoft is lower pressure for lower pay but it really hinders the talent pipeline.

The deal used to be a lower cost of living in a major coastal city, an amazing campus (it is seriously lovely), every engineer had their own office, serious job security, and an unbelievable health care plan.

Seattle exploded in price, they moved to open offices, Microsoft started doing mass layoffs, and they gutted the healthcare plan (by the time I left the main plan on offer was a high deductible with a miserable prescription formulary).

Hard to attract talent when there is no big differentiator.

Of course in the 90s the deal was work there 10 years retire a millionaire. Easy to attract talent when that is the offer ...

[rawgabbit]

I believe the author was referring to this https://www.propublica.org/article/microsoft-digital-escorts....

Microsoft hired Chinese engineers to manage US Department of Defense Azure VMs.

[axelriet]

Thanks. That reference is correct. The point is why those sessions were necessary because there is no reason, a-priori, to do manual touches on production systems, DoD or not.

[staticassertion]

> risks to national security

Microsoft is the go to solution for every government agency, FEDRAMP / CMMC environments, etc.

> People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true.

This I'm more sympathetic to. I really don't think his approach of "here's what a rewrite would look like" was ever going to work and it makes me think that there's another side to this story. Thinking that the solution is a full reset is not necessarily wrong but it's a bit of a red flag.

[kklisura]

At no point during the reading I got sense that he's suggesting something radical. Where specifically is he pointing out rewrite?

"The practical strategy I suggested was incremental improvement... This strategy goes a long way toward modernizing a running system with minimal disruption and offers gradual, consistent improvements. It uses small, reliable components that can be easily tested separately and solidified before integration into the main platform at scale." [1]

[1] https://isolveproblems.substack.com/p/how-microsoft-vaporize...

[staticassertion]

> The current plans are likely to fail — history has proven that hunch correct — so I began creating new ones to rebuild the Azure node stack from first principles.

> A simple cross-platform component model to create portable modules that could be built for both Windows and Linux, and a new message bus communication system spanning the entire node, where agents could freely communicate across guest, host, and SoC boundaries, were the foundational elements of a new node platform

Yes, I read that part as well and found it a bit confusing to reconcile with this one.

The vibe from my quotes is very much "I had a simple from-scratch solution". They mention then slowly adopting it, but it's very hard to really assess this based on just the perspective of the author.

He also was making suggestions about significantly slowing down development and not pursuing major deals, which I think again is not necessarily wrong but was likely to fall on deaf ears.

[axelriet]

Interesting point. The two stances are not contradictory. The end result is a new stack, so you are right saying that was the intent. However how you get there on a running system is through stepwise improvements based on componentization and gradual replacement until everything is new. Each new component clears more ground. I never imagined an A/B switch to a brand new system rewritten from scratch.

[outworlder]

> Microsoft is the go to solution for every government agency, FEDRAMP / CMMC environments, etc.

I've been involved with FEDRAMP initiatives in the past. That doesn't mean as much as you'd think. Some really atrocious systems have been FEDRAMP certified. Maybe when you go all the way to FEDRAMP High there could be some better guardrails; I doubt it.

Microsoft has just been entrenched in the government, that's all. They have the necessary contacts and consultants to make it happen.

> Thinking that the solution is a full reset is not necessarily wrong but it's a bit of a red flag.

The author does mention rewriting subsystem by subsystem while keeping the functionality intact, adding a proper messaging layer, until the remaining systems are just a shell of what they once were. That sounds reasonable.

[axelriet]

Thanks. That was exactly the plan. Full rewrites are extremely risky (see the 2nd System syndrome) as people wrongly assume they will redo everything and also add everything everyone always wanted, and fix all dept, and do it in a fraction of the time, which is delusional and almost always fail. Stepwise modernization is a proven technique.

[guardiangod]

As someone who had worked adjacent to the functionally-same components (and much more) at your biggest competitor, you have my sympathy.

Running 167 agents in the accelerator? My gawd that would never fly at my previous company. I'd get dragged out in front of a bunch of senior principals/distinguished and drawn and quartered.

And 300k manual interventions per year? If that happened on the monitoring side , many people (including me) would have gotten fired. Our deployment process might be hack-ish, but none of it involved a dedicated 'digital escort' team.

I too have gotten laid off recently from said company after similar situation. Just take a breath, relax, and realize that there's life outside. Go learn some new LLM/AI stuff. The stuff from the last few months are incredible.

We are all going to lose our jobs to LLM soon anyway.

[staticassertion]

> I've been involved with FEDRAMP initiatives in the past. That doesn't mean as much as you'd think. Some really atrocious systems have been FEDRAMP certified. Maybe when you go all the way to FEDRAMP High there could be some better guardrails; I doubt it.

I never said otherwise. I said that Microsoft services are the defacto tools for FEDRAMP. I never implied that those environments are some super high standard of safety. But obviously if the tools used for every government environment are fundamentally unsafe, that's a massive national security problem.

> Microsoft has just been entrenched in the government, that's all.

Yes, this is what I was saying.

> The author does mention rewriting subsystem by subsystem while keeping the functionality intact, adding a proper messaging layer, until the remaining systems are just a shell of what they once were. That sounds reasonable.

It sounds reasonable, it's just hard to say without more insight. We're getting one side of things.

[Spooky23]

FedRAMP means nothing. It’s a checkbox. National security stuff has a different standard.

[staticassertion]

It "means nothing" that the way that government systems get set up for government data is all using Microsoft tooling?

[Spooky23]

The tail wags the dog. GCP and Workspace have had better FedRAMP certs for ages.

[staticassertion]

I have no idea what you're talking about. This has nothing to do with having "better fedramp certs". If you are setting up fedramp or cmmc you will be heavily, heavily pressured and incentivized to do so with Microsoft tooling.

"Better" isn't relevant, which is my entire point. The reason people choose Microsoft isn't "it's better for this", it's because every consultancy out there, every government agency or affiliate, etc, is going to push Microsoft very very hard.

[Spooky23]

I’ve been in the space for 30 years. Nobody is pressuring anyone to buy Microsoft because of FedRAMP, and Microsoft is not even close to having any advantage with respect to FedRAMP vs their competitors.

FedRAMP is demonstration that the solution met some assessment of controls in alignment with NIST 800-53. As a checkbox, it’s almost as dumb as FIPS 140, and like FIPS, you need to asses risk for your implementation regardless of these things.

Microsoft wins deals because their product catalog is well engineered to incentivize bundled subscriptions that drive marginal adoption. The user facing products are better, Entra is generally right there, and that’s a pivot into many other scenarios that drive spend.

[lokar]

I think he did kind of point at the lack of seniority in the org, so I'm not sure he was trying to exaggerate with the titles.

I'm really struck that they have such Jr people in charge of key systems like that.

[markus_zhang]

Juniors love to hack out new things and in the mean time they can take the blame if needed, fair trade, won’t you say?

[nrds]

I've worked at both Microsoft and Google in the past 6 years and the notion that msft "Principal" is equivalent to goog L5 is crazy.

[dh2022]

Meaning Msft Principal is below L5? I got the same feedback from one of my friends who works at Google. She said quality of former MSFT engineers now working at Google was noticeably lower.

[nrds]

I mean imputed prestige within the organization. Being an L5 is nothing; it's the promote-or-fire cutoff at Google AFAIK. But being a Principal is slightly more than nothing; it's two levels above the promote-or-fire cutoff.

I mean, _now_, sure, I'd assume Microsoft Principals should be hired around L4 at Google. But that's just due to a temporary inbalance in the decline of legacy organizations. Give it a few years and it will even back out and msft 64 will be in the middle of L5 range like levels.fyi claims.

[compiler-guy]

L5 hasn't been the promote or fire cutoff at Google for perhaps a decade. L4 is the new L5, mostly because Google would have to pay L5s more, and it has been terrified of personnel costs for years.

But even so, an L5 at Google is basically a nobody as far as prestige or convincing other people to adopt your plan goes. Even L6 is basically just an expert across several mostly local teams. L7 is where the prestige gets going.

[zipy124]

I mean if you go by pay in the UK a Microsoft principle is equivalent to an L4 at Google if levels.fyi is too be believed....

[abtinf]

> risks to national security …really?

Really. Apparently the Secretary of War agrees with him.

[mpyne]

In fairness the SECWAR is hardly a computing expert.

But in this case the SECWAR has been properly advised. If anything it's astonishing that a program whereby China-based Microsoft engineers telling U.S.-based Microsoft engineers specific commands to type in ever made it off the proposal page inside Microsoft, accelerated time-to-market or not.

It defeats the entire purpose of many of the NIST security controls that demand things like U.S.-cleared personnel for government networks, and Microsoft knew those were a thing because that was the whole point to the "digital escort" (a U.S. person who was supposed to vet the Chinese engineer's technical work despite apparently being not technical enough to have just done it themselves).

Some ideas "sell themselves", ideas like these do the opposite.

[jacquesm]

> If anything it's astonishing that a program whereby China-based Microsoft engineers telling U.S.-based Microsoft engineers specific commands to type in ever made it off the proposal page inside Microsoft, accelerated time-to-market or not.

> It defeats the entire purpose of many of the NIST security controls that demand things like U.S.-cleared personnel for government networks, and Microsoft knew those were a thing because that was the whole point to the "digital escort" (a U.S. person who was supposed to vet the Chinese engineer's technical work despite apparently being not technical enough to have just done it themselves).

That is beyond bad. Proof of this?

[mpyne]

https://www.propublica.org/article/microsoft-digital-escorts...

[jacquesm]

Holy fuck. Ok, this will change things considerably for some companies I'm working with that had moved their stuff to Azure. Thanks. More than I can express on here.

[lmm]

Being compliant with the letter of the requirements at 1/3 of the cost is absolutely an idea that sells itself.

[CoastalCoder]

I'd like to suggest calling him SECDEF, not SECWAR.

IMHO the country should not capitulate to Trump's power grabs, even if Congress refuses to perform their oversight duties.

[mpyne]

I'm sympathetic to the viewpoint but I'm not in the habit of policing the names people use for themselves.

I've certainly done more than my fair share of jobs in the Navy where the office I was formally billeted to had long since ceased to actually exist as described due to office renamings. Often things as simple as a department section being elevated into a department branch and people using the new name even while they wait 1-2 years for the manpower records to be fixed and the POM process to cycle through for program resourcing. But still, seems hard to treat it as a crime at one level when no one blinked an eye at the lower level.

Maybe Congress will eventually step in, but in the meantime the American voters made their choice about who they want to run these agencies, so...

[cwillu]

The main title of the office is still “secretary of defense”, the executive order added a secondary title of the department and the office, it didn't replace the primary titles.

[jubilanti]

> the American voters made their choice about who they want to run these agencies

The American voters don't get to override the U.S. constitution. The American voters also voted in the U.S. Congress, which has the sole authority to name the department and title. My representatives have not voted to change the law. Do you not care about the rule of law?

> I'm not in the habit of policing the names people use for themselves.

I'm sure you think you're being clever, but this is such a bad faith argument.

[mpyne]

> Do you not care about the rule of law?

Of course I do. I hope the rest of my fellow Americans will someday care as much as I do about it. It's clearly not the case today.

But, is it illegal to refer to Secretary Hegseth as the SECWAR?

If so, would it be legal to refer to him as the SECDEF? After all, that isn't the formal term that Congress established his position as under 10 USC 113.

It's not hard to see all the cans of worms that emanate from the topic. I said already that this is Congress's purview, and they have had ample opportunity to put a stake in the ground on their position in response...

[jazzypants]

These agencies such as the Department of Defense, whose secretary is...?

The department's name is *legally* the Department of Defense. If they want to change it, they can go to Congress and do it the legal way. They have a majority. There's nothing stopping them except for their disregard for the sanctity of the law.

[gzread]

We could call him by what he does: SECMASSMURDERER

[asteroidburger]

The United States does not have a Secretary of War, and has not since 1947.

[7bit]

Uhm:

> The United States secretary of defense (SecDef), secondarily titled the secretary of war (SecWar),[b] is the head of the United States Department of Defense (DoD), the executive department of the U.S. Armed Forces, and is a high-ranking member of the cabinet of the United States.[8][9][10]

Wikipedia

[pinkmuffinere]

To be fair, it's not like Hegseth is a super high-signal source. Hegseth says lots of stuff, some of which are even true!

[wredcoll]

This was such a genuinely weird moment for me when reading the article.

"yadda yadda and then also the secretary of defence agreed it was bad"

I'm just reading along and going, "yeah that sounds really bad if a secretary level position is being cited... wait a second, isn't that actually the guy who is literally famous for being stupid??"

I never expected to be living through a real life version of "the emperor's new clothes", like, how is anyone quoting this guy about anything?

[UltraSane]

The problem is that what he writes is very plausible and explains a lot about why Azure is so unreliable and insecure. The author didn't mention the shameful way Microsoft leaked a Golden SAML key to Chinese hackers. This event absolutely was a threat to national security.

[dartharva]

Do you contest the fact that Microsoft royally fumbled OpenAI out of sheer incapability of providing what's supposed to be its core business despite having all deals in its favor? Because that's the most damning validation against Azure in recent times.

[dgellow]

If your reaction is emblematic of the way people reacted to his points internally that does give more credibility to his side of the story IMHO

[Hikikomori]

Chugging along? Very clear you're not a customer using Azure.

[whoamii]

For reference, author was a Senior Software Engineer, ie. mid-level engineer.

[markus_zhang]

He was from the kernel team, though. I always put their experience x2.

[whoamii]

Unless you’re still senior after 10+ years. Add to that switching companies and coming back same level…

[markus_zhang]

I don't disagree with you. I wish there were some good counter points from the Azure team. There was one from the Azure team in who commented on the article, but I feel that comment to be a bit weak.

[axelriet]

Life works in mysterious ways. Whoever you are, bring it on and prove any of my points wrong.

[whoamii]

Like the one where 1.5T in value went pfoof due to reasons you mentioned? I will let people judge whether your arguments are most likely, or whether this is bubble syndrome. Hint: there is a large distance between use of smart pointers and market effects.

[axelriet]

Hint: there is a small distance, almost zero, between your product being unreliable and losing your most flagship customer and the trust of the DoD.

Factor that there are limits to what I was willing to share in that story, limits that I’ll certainly not cross here.

Now, if you think that I’m wrong and that Microsoft found an entirely different way of shooting itself in the foot, I’d love to hear your perspective.

[small_model]

Yes it's easy to critique any large system or organisation, to then go over everyone's head and cry to the CEO and Board is snake like behaviour especially offering you self as the answer to fix it. OP will be marked as a troublemaker and bad team member.

[axelriet]

Maybe. That would be a dent in the shiny culture of trust Microsoft is proud to run on, though.

[axel479343]

The grudge is simple and doesn't detract one thing from a very well articulated blog: you do you job as an engineer of pointing out problems, even proposing solutions, and they fire you for doing exactly the job. It's infuriating enough just from reading it, idk how you can't see any legitimacy on what the guy is complaining. You have your right of free speech to complain about shitty jobs if you want, there's no honor bound to maintain silence here.

[sabedevops]

He might sound like he has a grudge but you sound like you’re personally invested. Shill?