Decisions that eroded trust in Azure – by a former Azure Core engineer

(isolveproblems.substack.com)

208 points | by axelriet 8 hours ago

21 comments

  • _pdp_ 1 minute ago
    The personal account makes a lot of sense, although I could easily see why the OP was not successful. Even if you are an excellent engineer, making people do things, accept ideas, and in general hear you requires a completely different skill altogether - basically being a good communicator.

    The second thing is that this series of blog posts (whether true or not, but still believable) provides a good introduction to vibe coders. These are people who have not written a single line of code themselves and have not worked on any system at scale, yet believe that coding is somehow magically "solved" due to LLMs.

    Writing the actual code itself (fully or partially) maybe yes. But understanding the complexity of the system and working with organisational structures that support it is a completely different ball game.

  • yoyohello13 1 hour ago
    I don't know if any of this is true, but as a user of Azure every day this would explain so much.

    The Azure UI feels like a janky mess, barely being held together. The documentation is obviously entirely written by AI and is constantly out of date or wrong. They offer such a huge volume of services it's nearly impossible to figure out what service you actually want/need without consultants, and when you finally get the services up who knows if they actually work as advertised.

    I'm honestly shocked anything manages to stay working at all.

    • giancarlostoro 7 minutes ago
      On the one hand I prefer Azure because unlike AWS, things have names I can easily remember and attribute to a problem I'm solving in my head. With AWS I can't remember what some of the things are unless I've been using it consistently, with Azure I don't even have to use a service, I'll remember the name for years.

      The UI though, could be improved. Still not as awful as AWS is.

    • ryoshu 33 minutes ago
      I’ve worked with their consultants and they were lovely. They hate Azure too.
      • everdrive 20 minutes ago
        I imagine that no one likes Azure.
    • ragall 49 minutes ago
      We migrated some services to AKS because the upper management thought it was a good deal to get so many credits, and now pods are randomly crashing and database nodes have random spikes in disk latency. What ran reliably on GCP became quite unpredictable.
      • SeriousM 23 minutes ago
        Interesting! We're using AKS with huge success so far, but lately our Pods are unresponsive and we get 503 Gateway Timeouts that we really can't trace down. And don't get me started on Azure Blob Tables...
        • ragall 6 minutes ago
          In our case this was only a month ago, and now we're stuck because management thought it was a good idea to sign a hefty spend commitment.
  • vintagedave 2 hours ago
    What are we reading here? These are extraordinary statements. Also with apparent credibility. They sound reasonable. Is this a whistleblower or an ex employee with a grudge? The appearance is the first. Is it? They’ve put their name to some clear and worrying statements.

    > On January 7, 2025… I sent a more concise executive summary to the CEO. … When those communications produced no acknowledgment, I took the customary step of writing to the Board through the corporate secretary.

    Why is that customary? I have not come across it, and though I have seen situations of some concern in the past, I previously had little experience with US corporate norms. What is normal here for such a level of concern?

    More, why is this public not a court case for wrongful termination?

    Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience?

    • bumblehean 3 minutes ago
      >Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience?

      IME, yes.

      I'm currently working as an SRE supporting a large environment across AWS, Azure, and GCP. In terms of issues or incidents we deal with that are directly caused by cloud provider problems, I'd estimate that 80-90% come from Azure. And we're _really_ not doing anything that complicated in terms of cloud infrastructure; just VMs, load balancers, some blob storage, some k8s clusters.

      Stuff on Azure just breaks constantly, and when it does break it's very obvious that Azure:

      1. Does not know when they're having problems (it can take weeks/months for Azure to admit they had an outage that impacted us)

      2. Does not know why they had problems (RCAs we're given are basically just "something broke")

      3. Does not care that they had problems

      Everyone I work with who interacts with Azure at all absolutely loathes it.

    • ZeroCool2u 2 hours ago
      In my experience Azure is full of consistency issues and race conditions. It's enough of an issue that I was talking about new OpenAI models becoming available via Bedrock on AWS and how convenient that was since I wouldn't have to deal with Azure and my colleague in enterprise architecture went on an unprompted rant about these exact issues. It's not the first time something like this has happened and I've experienced these issues first hand, so yes. I'd say reliability is a critical issue for Azure and it hasn't gotten better each time I've gone back to check.
    • rando1234 54 minutes ago
      I recall seeing some pretty damning reports from a security pentester that was able to escape from a container on Azure and found the management controller for the service was years old with known critical unpatched vulnerabilities. Always been a bit sceptical of them since then
    • convexly 40 minutes ago
      Large orgs make decisions that prioritize short-term metrics over long-term quality all the time and nobody tracks whether those tradeoffs actually paid off. The decision to ship fast and fix later sounds reasonable in a meeting setting until articles like this surface and the reality comes through clearly.
    • zipy124 1 hour ago
      Yes it is that unreliable. Even when given free credits, I would rather pay for the offerings from Amazon/Google.
    • chasd00 34 minutes ago
      Yeah I thought that was extreme. An engineer going to the board of any corporation let alone Microsoft is not normal or customary IME. That could explain why they got no response.
    • bigbuppo 2 hours ago
      The CEO is accountable to the board. If they are derelict in their obligations to the company, that's where you need to raise a stink so they can fix it.
      • ohyoutravel 54 minutes ago
        Well, yeah, that’s what a board does, but I think the issue is whether it is customary to go to the board directly in this situation. The answer is a resounding NO. Very odd, but cool idea and approach.
    • lokar 1 hour ago
      He is, I think, Swiss, perhaps a cultural difference?
    • pRusya 1 hour ago
      Azure is when you have a different version of the same product/api in each region.
  • OldOneEye 1 hour ago
    Some previous colleague of mine has to work with Azure on their day to day, and everything explained in this article makes a lot of sense when I get to hear about their massive rantings of the platform.

    12 years ago I had to choose whether to specialize myself in AWS, GCP or Azure, and from my very brief foray with Azure I could see it was an absolute mess of broken, slow and click-ops methodology. This article confirms my suspicions at that time, and my colleague experience.

  • pRusya 6 hours ago
    It's a nice read. Thank you for sharing this.

    > Microsoft, meanwhile, conducted major layoffs—approximately 15,000 roles across waves in May and July 2025 —most likely to compensate for the immediate losses to CoreWeave ahead of the next earnings calls.

    This is what people should know when seeing massive layoffs due to AI.

  • lokar 1 hour ago
    This reads pretty bad, and I believe it was. I worked on (and was at least partly responsible for) systems that do the same thing he described. It took constant force of will, fighting, escalation, etc to hold the line and maintain some basic level of stability and engineering practice.

    And I've worked other places that had problems similar to the core problems described, not quite as severe, and not at the same scale, but bad enough to doom them (IMO) to a death loop they won't recover from.

  • nope1000 4 hours ago
    > The direct corollary is that any successful compromise of the host can give an attacker access to the complete memory of every VM running on that node. Keeping the host secure is therefore critical.

    > In that context, hosting a web service that is directly reachable from any guest VM and running it on the secure host side created a significantly larger attack surface than I expected.

    That is quite scary

    • xorcist 1 hour ago
      Scary is the understatement of the day. I can't imagine the environment where someone think that architecture is a good idea.
  • schlauerfox 3 hours ago
    "For fiscal 2025, Microsoft CEO Satya Nadella earned total pay of $96.5 million, up 22% from a year earlier." -CNBC.com

    and

    "I also see I have 2 instances of Outlook, and neither of those are working." -Artemis II astronaut

    • tantalor 2 hours ago
      > 2 instances of Outlook

      That's 2 too many.

      • bigbuppo 2 hours ago
        They should have used the third outlook they didn't know about... Outlook, Outlook (new), and the well-hidden Outlook (classic) that actually works.
        • cyanydeez 1 hour ago
          That outlook was part of the ablative outlook armor thats suppose to burn off on reentry
  • ludwigvan 1 hour ago
    I had the misfortune of having to use Azure back in 2018 and was appalled at the lack of quality, slowness. I was in GitHub forums, helping other customers suffering from lack of basic functionality, incredible prices with abysmal performance. This article explains a lot honestly.

    Google’s Cloud feels like the best engineered one, though lack of proper human support is worrying there compared to AWS.

    • reddozen 36 minutes ago
      GCP's support is abysmal. Our assigned customer support agent has changed 3 times in as many months. it's really a dice roll if our quota increase requests are even acknowledged or we can get clarification on undocumented system limitations.
  • Anon1096 1 hour ago
    The post is so dramatized and clearly written by someone with a grudge such that it really detracts from any point that is trying to be made, if there is any.

    From another former Az eng now elsewhere still working on big systems, the post gets way way more boring when you realize that things like "Principle Group Manager" is just an M2 and Principal in general is L6 (maybe even L5) Google equivalent. Similarly Sev2 is hardly notable for anyone actually working on the foundational infra. There are certainly problems in Azure, but it's huge and rough edges are to be expected. It mostly marches on. IMO maturity is realizing this and working within the system to improve it rather than trying to lay out all the dirty laundry to an Internet audience that will undoubtedly lap it up and happily cry Microslop.

    Last thing, the final part 6 comes off as really childish, risks to national security and sending letters to the board, really? Azure is still chugging along apparently despite everything being mentioned. People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true.

    • kraemahz 1 hour ago
      AWS and Google Cloud are both huge and are significantly better in UX/DX. My only experience with Azure was that it barely worked, provided very little in the way of information about why it didn't. I only have negative impressions of Azure whereas at least GC and AWS I can say my experiences are mixed.
    • lokar 1 hour ago
      I think he did kind of point at the lack of seniority in the org, so I'm not sure he was trying to exaggerate with the titles.

      I'm really struck that they have such Jr people in charge of key systems like that.

    • abtinf 26 minutes ago
      > risks to national security …really?

      Really. Apparently the Secretary of War agrees with him.

    • sabedevops 1 hour ago
      He might sound like he has a grudge but you sound like you’re personally invested. Shill?
    • jiggawatts 1 hour ago
      > People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true.

      Or… you’ve just normalised the deviation.

      One of the few reliable barometers of an organisation (or their products) is the wtf/day exclaimed by new hires.

      After about three or four weeks everyone adapts, learns what they can and can’t criticise without fallout, and settles into the mud to wallow with everyone else that has become accustomed to the filth.

      As an Azure user I can tell you that it’s blindingly obvious even from the outside that the engineering quality is rock bottom. Throwing features over the fence as fast as possible to catch up to AWS was clearly the only priority for over a decade and has resulted in a giant ball of mud that now they can’t change because published APIs and offered products must continue to have support for years. Those rushed decisions have painted Azure into a corner.

      You may puff your chest out, and even take legitimate pride in building the second largest public cloud in the world, but please don’t fool yourself that the quality of this edifice is anything other than rickety and falling apart at the seams.

      Remind me: can I use IPv6 safely yet? Does it still break Postgres in other networks? Can azcopy actually move files yet, like every other bulk copy tool ever made by man? Can I upgrade a VM in-place to a new SKU without deleting and recreating it to work around your internal Hyper-V cluster API limitations? Premium SSDv2 disks for boot disks… when? Etc…

      You may list excuses for these quality gaps, but these kinds of things just weren’t an issue anywhere else I’ve worked as far back as twenty years ago! Heck, I built a natively “all IPv6” VMware ESXi cluster over a decade ago!

    • irishcoffee 1 hour ago
      > The post is so dramatized and clearly written by someone with a grudge such that it really detracts from any point that is trying to be made, if there is any

      I guessed that from the title on the main hn page. Glad to see it confirmed.

  • gnabgib 3 hours ago
    Title: How Microsoft Vaporized a Trillion Dollars
  • abtinf 33 minutes ago
    So this is why GitHub is having so many problem…
  • Bjartr 4 hours ago
    What a fascinating view into how the sausage is made
  • acedTrex 2 hours ago
    This is an insanely blunt look into some serious issues with microsoft.
  • ok123456 30 minutes ago
    New trollaxor dropped.
  • andrewstuart 2 hours ago
    Any complex system - and these cloud systems must be immensely complex - accumulate cruft and bloat and bugs until the entire thing starts to look like an old hotel that hasn’t been renovated in 30 years.
    • lll-o-lll 46 minutes ago
      It’s not inevitable. Absolutely this is true without significant effort, but if you’ve been around the traps for long enough (in enough organisations), you get to see that the level of quality can vary widely. Avoiding the mud-pit does require a whole org commitment, starting from senior leadership.

      This story is more interesting, in my opinion, in how quickly things devolved and also how unwilling the more senior layers of the org were to address it. At a whole company level, the rot really sets in when you start to lose the key people that built and know the system. That seems to be what’s happening here, and it does not bode well for MS in the medium term.

  • axelriet 8 hours ago
    A former Azure Core engineer’s 6-part account of the technical and leadership decisions that eroded trust in Azure.
    • jacquesm 1 hour ago
      Why do you speak about yourself in the third person?

      Also, after this:

      https://news.ycombinator.com/item?id=20341022

      You continued to work at Microsoft and now there is this takedown?

      I'm no friend of MS (to put it very mildly) but it seems to me your story is a bit inconsistent as well as the 7 year break between postings.

      • netruk44 25 minutes ago
        > Why do you speak about yourself in the third person?

        When you submit a link to HN, there is an entry field for a comment.

        It does not really describe what the comment is used for. For links, it simply gets added as the first comment.

        Someone who is unfamiliar with the submission process may assume this comment should describe what they are submitting, and not format it like a regular user comment.

        Then it gets posted as the first comment and tons of people downvote it, jumping to the conclusion that the weird summary comment is from an AI, and not the submitter describing their own submission.

    • AceJohnny2 2 hours ago
      I downvoted this comment for sounding like a summarizing LLM, not adding anything substantial beyond the title of the post, before realizing you were the poster and author.
    • ninininino 1 hour ago
      What's your assessment of AWS and GCP? Do you think it's likely they suffer from some of the same issues (eg the manual access of what should be highly secure, private systems, the instability, the lack of security)?
      • rybosome 8 minutes ago
        As a former GCP engineer, no, the systems are not generally unstable or insecure.

        There is definitely manual access of data - it requires what was termed “break glass” similar to the JIT mechanism described by the author. However, it wasn’t quite so loose; there were eventually a lot of restrictions on who could approve what, what access you got after approval, and how that was audited.

        It was difficult to get into the highest sensitivity data; humans reviewed your request and would reject it without a clear reason. And you could be 100% sure humans would review your session afterwards to look for bad behavior.

        I once had to compile a large list of IP addresses that accessed a particular piece of data to fulfill a court order. It took me days of effort to get and maintain the elevated access necessary to do this.

        I have a lot of respect for GCP as an engineering artifact, but a significantly less rosy opinion of GCP as an organization and bureaucratic entity. The amount of wasted effort expended on engaging with and navigating the bureaucracy is truly mind-boggling, and is the reason why a tiny feature that took a day to code could take months to release.

  • brcmthrowaway 2 hours ago
    What an epic takedown.

    Microsoft should have promoted this guy instead of laying him off.

    Did Microsoft really lose OpenAI as a customer?

  • pavlov 1 hour ago
    The first couple of paragraphs felt like a parody of a guy who goes to a diner and gets upset the waitress doesn’t address him as Dr.

    It didn’t get any better.

    • lokar 59 minutes ago
      His writing style is fairly over the top (he is Swiss, and I have seen this before, but not most of the time), but most of the technical content seems true to me.