Tigris rebuilds Git's packfile format to run directly on object storage

The problem: Git was designed for disk, not for buckets

Git's internal format starts from a simple premise: everything lives on a local filesystem, and the kernel can mmap the pack files (packfiles) to treat disk content as memory. Reading a specific object costs nanoseconds because the filesystem cache handles it.

Object storage doesn't have that luxury. A GetObject call costs, at minimum, 10 milliseconds of network round-trip, orders of magnitude slower than local disk access. That difference is what led Xe Iaso, an engineer at Tigris Data, to document in technical detail on the company's blog why she abandoned the obvious approach (using a filesystem shim on top of object storage) and rewrote from scratch the packfile format used by objgit, the open source Git server that Tigris is building on top of its own object storage infrastructure.

Market context matters here: as the author herself notes, several companies have launched hosted Git products in recent months, and the reason is the same for all of them: running Git as a managed, elastic, and cheap service requires decoupling the storage format from the "local filesystem" assumption that Git has carried since 2005.

Why the range request alone doesn't solve it

A packfile stores tens of millions of objects (blobs, trees, commits) compressed and concatenated into a single binary file, with a separate .idx index that points to the offset of each object. In theory, it would be enough to make an HTTP range request to fetch only the bytes of a specific object inside the bucket, without downloading the entire packfile.

The problem is that Git's default index stores the starting offset of each object and its uncompressed size, but not its compressed size. Without knowing how many bytes the object occupies in the file, it's not possible to build a precise range request: you know where to start, but not where to stop. That detail alone makes it unviable to use conventional packfiles directly against an S3 bucket efficiently.

The solution: separating data from metadata, like on an audio CD

The way out Iaso found was to completely separate data from metadata, inspired by the .bin/.cue format used for backing up audio CDs and multi-session game discs (Dreamcast, Xbox 360). In objgit:

• Objects are concatenated, compressed with zstd, into an objects.bin file of up to 128 MiB;
• A separate objects.cue file stores fixed-size records (58 bytes each) with the SHA-1 hash, type, compression algorithm, offset within the .bin, compressed size, uncompressed size, and the reference to the base object in case of a delta.

With the compressed size explicit in the .cue, the client calculates exactly the range request needed and requests only that byte range from the bucket, instead of downloading the entire file. The author also changed how deltas are stored: instead of being appended to the end of the original object (as in Git's classic packfile), each delta becomes its own record in the .cue, avoiding having to read the entire base object just to reach a specific revision.

Downloading while reading: racing the beam

To reduce latency in practice, objgit uses a strategy that the author herself describes as racing the beam: whenever the Git library requests an object from a packfile, objgit starts downloading the entire .bin to a temporary folder in the background. While the download hasn't reached the position of the requested object, the client fires targeted range requests directly at the bucket; as soon as the background download passes that offset, the range requests stop being necessary.

In practice, this means the latency cost is concentrated only on the part of the file not yet downloaded, and the rest arrives "for free" through the sequential download running in parallel.

What changes for those who build infrastructure

For those who maintain CI/CD pipelines, ephemeral runners, or internal Git servers, the proposal tackles a real point of friction: today, running Git on object storage generally means mounting a bucket via a FUSE/filesystem gateway, which inherits all the slowness object storage has for small, random operations. A native object storage format, like the one Tigris describes, opens the door to:

• Cheaper partial clones: fetching only the objects needed for a shallow checkout, without needing a filesystem shim in the path;
• Elastic Git servers: with no local disk state per repository, the server can scale horizontally because the "disk" is the bucket;
• More direct integration with cloud-native pipelines: CI runners that already read and write to S3/Azure Blob could, in theory, interact with Git objects without an extra synchronization layer.

The open point, which the author herself implicitly acknowledges, is compatibility: this .bin/.cue format is proprietary to objgit, not a standard Git packfile, so any server that uses it needs a translation layer to speak the Git protocol with regular clients (the git command-line tool still sees an ordinary Git repository from the outside). The justification given is that, since Git is a distributed version control system, every clone already contains the entire history, so recreating the storage format on the server side is a low-risk bet: if it goes wrong, you just repack the data from any existing clone.

The project is still under development and is open source, which means teams that currently run Gitea, Gitolite, or equivalent self-hosted solutions on top of cloud volumes have a concrete case study of how to decouple Git from a traditional filesystem, something relevant for those operating their own infrastructure in Brazil and paying for IOPS or block storage that's more expensive than plain object storage.