GitHub redesigns Git storage to support 5,000 simultaneous AI agents

GitHub rebuilt the storage layer that holds every repository, sustaining in internal tests 5,000 AI agents writing at the same time to a single repository, at 7,200 pushes per second.

GitHub rebuilt the storage layer that holds every repository, sustaining in internal tests 5,000 AI agents writing at the same time to a single repository, at 7,200 pushes per second.

The bottleneck AI agents exposed in Git

Today, GitHub's busiest repositories receive at most a few dozen writes per second, in short bursts. According to GitHub, as more of this work gets done by agents running in parallel around the clock, the company expects these bursts to become a sustained rate over the next year. One specific repository, shown by GitHub in an internal chart, had its daily traffic multiply by 4 between March and May, and the curve kept growing exponentially after that.

The problem isn't just the volume of git push. Each push generates cascading reads: CI pipelines fire, deployment systems pull the new commit, and other agents fetch the code for context before continuing to work. This amplification effect makes read traffic grow alongside write traffic, multiplying the pressure on the infrastructure that stores and serves repositories.

It was this scenario that led GitHub to redesign, under the hood and with the platform live, the storage layer that underpins every Git repository on the site. The announcement came at GitHub Universe, the week before the post published on November 5, 2026 on the company's engineering blog, signed by Karthik Raman and Brian Celenza (source).

Project Rubicon: two principles, a rebuild in production

The work began in May 2026, right after GitHub completed the migration of 99.99999% of repositories to Azure, a step covered in an earlier post in the same series. The team named the project "Rubicon."

We called it "Rubicon," a nod to the point of no return the industry crossed with agentic development.

Karthik Raman and Brian Celenza, GitHub Engineering

Two principles guided the design. The first is minimizing coordination: since Git objects are immutable and named by their own content, two writers can write them at the same time without needing to agree on anything. Only the reference update, the pointer that says where a branch is, requires agreement between writers, and that part is tiny compared to the rest of a push.

The second principle is decoupling storage from compute. Storing data, serving requests, and performing repository maintenance stop being tied to the same machine and each starts scaling on its own.

What the architecture looks like: Spokes, virtual replicas, and Ref-CAS

At the entry point, nothing changes for Git users: requests from Git clients, the API, and the web interface still arrive through Spokes, which handles routing and load balancing. It's the data layer below it that was rebuilt.

Spokes routes each request to one of a small group of virtual replicas designated for that repository. Virtual replicas are a single fleet of disposable compute: each one's local disk acts as a cache, rebuildable at any moment, and can serve reads, writes, or both.

The repository's durable state lives in two separate places:

• Git objects (packs and auxiliary read-acceleration data) live in Azure Blob Storage.
• References and the pack manifest (the list of packs that make up the repository) live in a dedicated replicated log service, named Ref-CAS ("Reference Compare-and-Swap").

In the write flow, a virtual replica indexes the received pack, uploads it to Blob Storage, and only then records a transaction in Ref-CAS, which lists the pack in the manifest and updates the references. In the read flow, the replica checks the reference version with Ref-CAS, downloads from Blob Storage only the packs missing from its cache, and serves the response from there.

Why Git needed a dedicated service just for references

The project's most counterintuitive choice was not putting everything in Blob Storage, even though object storage is cheap, durable, and scales well when traffic spreads across many keys. The problem is the access pattern of references: on a busy branch, multiple writes compete to update the same pointer at the same time, and that isn't object storage's strong suit.

GitHub tested two approaches before arriving at Ref-CAS. The first indexed all of a repository's references in a single blob, with each update being a conditional write against it: correct and cheap for reads, but the write ceiling was one write at a time against that single blob, which in practice capped throughput at hundreds of writes per second.

The second approach split references across several segment blobs, with each reference falling into a segment by hash. This raised the write ceiling, but moved the problem elsewhere: every read now had to check all segments, and became as slow as the slowest segment, precisely when reads outnumber writes by orders of magnitude.

Neither met all the requirements, and that's where Ref-CAS was born: a replicated log service dedicated to references, with each update being a compare-and-swap confirmed by quorum. A transaction locks only the references it touches, checks whether each one still has the value the writer expects, and only becomes visible once a majority of the log's replicas have recorded it. Since agents mostly write to their own branches, most updates don't contend for the same reference and replicate in parallel.

Each commit in Ref-CAS gets a log sequence number (LSN), and the repository's version is the LSN of the most recent commit. On a read, the virtual replica sends the version it already has: if it's still current, the response carries no reference at all, and an unchanged repository costs a single comparison. With this, GitHub says it can transact reference state on its own in the tens of thousands of operations per second.

Compaction out of the way of readers

Scaling writes creates a second-order problem: every push adds a new pack to Blob Storage, and although Git can serve a repository from several packs, each additional pack makes lookups a bit more expensive. In GitHub's old architecture, compaction ran on the same replicas that served traffic, and had to run on all of them.

In the new architecture, compaction runs entirely in the background, on dedicated nodes that work directly against durable storage, using Ref-CAS to notify virtual replicas about the updated state. This frees up virtual replicas' CPU and memory almost entirely for serving requests.

Each compaction node takes, via lease, ownership of a subset (shard) of a repository's packs, which allows compaction to scale horizontally. The node accumulates a shard's new packs in an "open segment" until it hits a limit of count, size, or age; then it seals the segment, combines the packs into a smaller, larger set, verifies that the new packages contain exactly the same objects as the originals, publishes the result, and only then removes the old packs from the manifest.

The process runs in parallel across many shards at once, so adding shards and nodes increases compaction capacity. The result, according to GitHub, is a pack count that stays practically stable even as push volume rises.

The numbers from the internal test

In internal tests, GitHub says it sustained 5,000 simultaneous agents working on a single repository, sending 7,200 pushes per second, while keeping read latency consistently low. That's a jump of at least two orders of magnitude from the dozens of writes per second that today define a busy repository on GitHub.com, and it was this capacity target that the company set before designing Rubicon: not planning for the next increment, but handling a scenario a hundred times larger than the current one.

What changes for developers

For those who only use GitHub through the CLI or the interface, Git is still Git: git push, git fetch, git branch work the same way, with the same branch protections and reviews as always. The relevant change isn't in commands, it's in guarantees: repositories that today struggle with CI queues or commits lagging due to write contention have, in GitHub's proposal, a much higher throughput ceiling ahead, without needing any special migration.

This matters mainly for teams that already run AI agents opening PRs, reviewing code, or automating merges in parallel on the same repository, the scenario that motivated the rebuild. A small project today can, in GitHub's own words, "pick up a fleet of agents overnight" without needing special treatment to handle it. It also applies to large monorepos that concentrate multiple services and teams in a single repository: it's exactly this kind of repository where write contention and pack accumulation weigh most heavily in the old architecture.

What's still open

GitHub's post lists three future fronts: evolving the architecture further (reading only the needed object instead of the whole pack, partial replicas, global replicas with bounded staleness, scaling to zero when a repository goes idle), making agent operations cheaper (working without a full clone, a multi-file atomic commit API, real-time event streams), and continuing to contribute to Git upstream itself (lighter partial clones, removing scale bottlenecks that still exist in Git's code, faster history lookups via commit-graph).

None of these fronts has an announced delivery date, and GitHub hasn't disclosed, in this post, when the new architecture will serve 100% of GitHub.com repositories.

Source 1: The Github Blog (https://github.blog/engineering/architecture-optimization/inside-githubs-new-git-storage-architecture/)

Inside GitHub's new Git storage architecture - The GitHub Blog

Karthik Raman & Brian Celenza

November 5, 2026

|

16 minutes

Share: