GitHub 公佈 2026 年 9 月服務可用性報告

Our last update covered content through August. We did not publish another in September while other public content addressed the August 17 outage. This edition covers progress over the past two months.

The short version of August and September: we reduced shared-database pressure, added overload controls, created headroom for GitHub Actions, expanded failure containment for pull requests, and moved more production workloads to Azure. Significant incidents, a capacity pause, and a missed migration target show that the work is not complete.

By October 7, 120 of 160 production services were deployed in Central US, and 13 of 51 primary MySQL databases had moved. Azure also handled 65.4% of read traffic for GitHub’s core application, 48.3% of Git read traffic, and 80.2% of stream-processing traffic. These percentages measure different workloads and should not be added.

We paused Central US deployments and traffic increases on September 11 because capacity was uneven across availability zones. Permanent ramps resumed September 18 after temporary relief, but durable capacity work continues, with a goal of unlocking all available capacity by the end of October. We missed our aggressive September 30 goal to deploy every production service there. During a planned database failover, an automated emergency process disconnected Central US database copies and disrupted customers in that region. We disabled that process during planned handoffs while improving recovery.

Outside the regional migration, 24 authentication-core tables moved off GitHub’s oldest shared database, removing approximately one million queries per second from its replicas. The database cluster supporting issues and pull requests removed approximately 980 million unnecessary edit-history rows, creating an estimated 1.8 terabytes of recoverable storage. Weekly deadlocks fell approximately 77% from their June peak, and continuous cleanup now prevents the same unbounded growth from returning.

Half of sampled pull request traffic can now continue with limited functionality when a shared dependency fails. Production results showed several read operations remained available during a shared-database incident. Database traffic routing can also add capacity automatically during spikes; one group expanded from 14 to 19 instances during a September 23 increase in pull request activity. A production overload control now lets responders protect higher-priority API and web workflows by rejecting lower-priority requests before broader saturation. Operator documentation and training are complete.

GitHub Actions also gained operating headroom. Production changes reduced measured Redis operations by approximately 28% and relative CPU by approximately 30%. A separate infrastructure cache reduced Redis bandwidth by 84%, average response time by 66%, median response time for the primary inventory request by 96%, and queued requests by 95%.

The August 6 actions outage, August 17 github.com outage, September failover disruption, and controlled capacity pause are reminders that measured improvements matter only when they reduce customer risk under real conditions. The principle remains the same: availability, then capacity, then features.

September 13 09:16 UTC (lasting 1 hour and 28 minutes)

What happened?

Impact summary

Duration: approximately 2 hours

Customers may have seen requests fail, run slowly, or remain pending when opening or updating issues and pull requests, calling APIs, creating access tokens, starting GitHub Actions workflows, or signing up.

~96% of the customer requests creating issues through the web interface were impacted during the incident window. In addition, 90% of new customer signup and ~4% of actions workflows also experienced intermittent failures.

What went wrong and why?

A background data-cleanup job increased load on a shared database. The job had a safeguard designed to slow its writes when database replicas fell behind, but the safeguard did not account for incoming load on the primary database server. As a result, the job continued running until the primary reached its connections limit. When the primary ran out of available connections, requests that depended on it could not complete reliably.

Two other factors increased the impact and delayed recovery. First, there was no quick timeout on these database calls, so request handlers waited on the stalled database instead of failing fast. Second, a retry loop around token creation kept resending writes that were already failing, which held the database saturated rather than letting it recover.

How did we respond?

1. A background data-cleanup job began writing to a shared database.
2. The job’s safeguard monitored replica lag but did not account for load building on the database’s primary server.
3. The primary ran out of available connections, causing requests across dependent services to fail or run slowly.
4. Automated monitoring detected the incident, and we traced the failures to load on the shared database.
5. We reduced internal load and paused the background job.
6. The database recovered, and error rates across the affected services returned to normal.

How are we making incidents like this less likely or less impactful?

• We are implementing additional safeguards to improve database resilience and service availability. These measures will better control background processing based on available database capacity, reduce unnecessary retries during failures, and limit the impact of slow database requests.
• We are also improving service startup behavior to ensure that temporary database connectivity issues do not unnecessarily affect service recovery.

September 28 21:16 UTC (lasting 52 minutes) 

What happened?

Impact summary 

Duration: approximately 2 hours and 45 minutes 

This incident impacted Copilot code review across github.com and GHEC data-residency regions. Affected customers saw Copilot remain in a reviewing state without a review appearing, with delays ranging from ~30 minutes to a max of 2 hours. 

Some reviews on github.com continued to complete normally during the incident, while reviews in data-residency regions did not. Some customers still received delayed error messages for requests made during the incident. 

What went wrong and why?

Copilot code review uses a separate service to run reviews and 2 versions of the results are returned: a limited copy that excludes the review comments and a full copy that includes the completed review. Copilot code review posts reviews only from the full copy. A change to that service began sending the full copy only for some customers. For other customers, only the limited copy was sent.

As a result, Copilot code review could not build a review, but reported that the job had completed successfully. The missing review results were not detected due to lack of monitoring. The incident was detected as some of the reviews remained incomplete for more than 30 minutes, triggering an alert to first responders.

We mitigated the incident by reverting the change.

How did we respond?

1. A change to the service that runs Copilot code reviews began rolling out, and review results stopped appearing on pull requests. The change reached every environment.
2. Reviews began timing out, triggering an alert to the Copilot code review team. We declared an incident.
3. We traced the failures to the change and began reverting it.
4. While the revert completed its required checks, we posted the incident on the GitHub Status page.
5. The revert then reached all the affected regions. We resolved the incident and advised customers to request affected reviews again.

How are we making incidents like this less likely or less impactful?

• We are strengthening the reliability of Copilot code reviews by adding safeguards throughout the review and delivery process. These improvements will help ensure that completed reviews are delivered successfully, allow failed reviews to recover automatically, and provide earlier notification when issues occur.
• We are also improving our monitoring and deployment processes so that problems can be identified and addressed more quickly.

Follow our status page for real-time updates on status changes and post-incident recaps. To learn more about what we’re working on, check out the engineering section on the GitHub Blog.

The post GitHub availability report: September 2026 appeared first on The GitHub Blog.