GitHub CTO apologises to developers after one of the biggest outages
GitHub experienced a major outage lasting nearly eight hours due to infrastructure scaling failures and a surge in traffic. The company identified a combination of misconfigured autoscaling policies and a retry bug in Visual Studio Code as the primary causes for the service disruption.
Why it matters
As a critical platform for global software development, GitHub's reliability issues highlight the challenges of scaling infrastructure to meet massive increases in developer activity.
GitHub has apologised for the outage that took its platform down for 7 hours and 47 minutes on August 17, and the explanation is unusually blunt. Nobody pushed bad code. Nobody fumbled a configuration change. The platform simply ran out of room.In a blog post published on August 20, Vlad Fedorov wrote that traffic hit a new peak and a critical infrastructure component in GitHub’s Central US data centre failed to scale with it. Authentication buckled first, and the pressure spread outward through github.com, Actions, APIs, pull requests, issues and Copilot. “If you were trying to ship software that day, we let you down,” the post reads.What caused the GitHub outage: an autoscaling policy watching the wrong thingThe root cause analysis points at an Istio sidecar reaching its concurrency limit while the autoscaling policy monitored the host service instead. Load balancers in Central US saturated.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in