Microsoft Reveals Months-Old Bug Caused Global 365 Outage
Share
Microsoft has attributed last week’s extensive global Microsoft 365 outage to a software bug introduced in July, which triggered a cascading failure across its networking gateway servers. The nearly 24-hour disruption on 12th November left more than 80% of these critical servers unresponsive, severely impacting services such as Outlook, Exchange Online, Teams, and Copilot worldwide.
In a preliminary post-incident report shared within the Microsoft 365 Admin Center, the company identified three contributing factors: a previously undetected software bug, a surge in traffic following a Microsoft Teams failover, and reduced server capacity. The incident began at 11:20 AM ET on Thursday, 12th November, and services were not fully restored until 11:10 AM ET the following day.
Latent Bug and Cascading Failure
The latent bug was part of an update deployed to Microsoft 365 networking gateway servers in July. Although the update had operated without issue for over three months, Microsoft explained that its standard load testing failed to detect the flaw. This was because testing environments used less powerful hardware with fewer cores than production servers, meaning the bug was worsened on machines with more processing capabilities.
The bug was eventually triggered approximately 15 hours before the widespread outage. This occurred when Microsoft Teams automatically redirected traffic to the networking infrastructure typically used by Exchange Online. This increased traffic, coupled with a pre-existing 14% reduction in gateway server capacity due to an unrelated issue, created the precise conditions for the software defect to manifest.
As servers became overwhelmed and unresponsive, Microsoft’s systems began redirecting users to other available servers, which in turn also failed under the additional load. Within two hours, the defect had spread across Microsoft’s fleet, rendering over 80% of servers unresponsive at the peak of the disruption. Safeguards designed to prevent regional spread also failed, contributing to the worldwide impact.
While services like Teams, OneDrive, and SharePoint could switch to alternative infrastructure and recover earlier, Exchange Online could not. This prolonged email disruption for users. Recovery efforts were challenging, as restarting affected servers immediately overwhelmed them with backed-up traffic.
Remediation and Ongoing Investigation
After identifying the root cause, Microsoft engineers developed and deployed a fix, stabilising services throughout Friday, 13th November. Microsoft has since increased server capacity, enhanced monitoring, and implemented new safeguards to prevent similar cascading failures from traffic redirection.
Charles Lamanna, Microsoft’s Executive Vice President of Copilot, Agents, and Platform, issued an apology via X (formerly Twitter), stating, "We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations." The investigation is ongoing, with a final post-incident report anticipated later this week.




Leave a Reply