MaxText cuts distributed LLM training traffic by up to 65x

- Reading
- 6 min
- Tables
- 2
- Code
- 1
On November 19, Google published details of Streaming DiLoCo in MaxText, a technique that allows training LLMs on TPUs spread across different datacenters without the network becoming the training bottleneck.
The problem: a single datacenter is no longer enough
Training a state-of-the-art LLM traditionally requires a monolithic cluster: all accelerators in the same place, connected by extremely high-bandwidth interconnects, synchronizing gradients at every step. The problem is that scaling a single cluster like this runs into physical limits and cost. The obvious workaround, connecting multiple compute slices spread across different datacenters, runs into another limit: the wide area network (WAN) between these datacenters is too slow and has too much latency for standard data parallelism, which depends on synchronizing gradients globally at every step.
In a post published on November 19 on the Google Developers Blog, teams from Google DeepMind and Google Research detailed how MaxText, an open source LLM training framework in JAX maintained by Google, implements Streaming DiLoCo to solve exactly this bottleneck. The proposal: train across multiple metros (cities, regions, datacenters) without the network becoming the limiting factor.
How DiLoCo avoids synchronizing at every step
DiLoCo (Distributed Low-Communication), originally described in a Google DeepMind paper, starts from a simple idea: instead of synchronizing all devices at every step, it divides training into M independent groups, called learners. Each learner runs a local optimizer (usually AdamW) for H steps without exchanging anything with the other groups.
The algorithm operates on two layers:
- Inner loop (local): each learner trains on its own for H steps (in Google's experiment, H ranged from 37 to 65), using the fast internal interconnect of its own slice.
- Outer loop (global): every H steps, the learners compute the difference between the current weights and the weights at the start of the interval, perform an all-reduce of these differences over the network, and apply an outer optimizer (SGD with Nesterov momentum) to update the global weights.
Communicating orders of magnitude less often than traditional synchronous training, DiLoCo had already demonstrated comparable convergence in previous work. But the "vanilla" version has a practical problem: every H steps, all learners need to transfer the delta of the entire model at once, which creates a huge traffic spike and stalls the pipeline on a narrow network.
The Streaming trick: fragment the model, not the time
Streaming DiLoCo solves this spike by fragmenting the model's parameters into P disjoint pieces. Over the course of H steps, each fragment is synchronized exactly once, keeping the same total communication volume as regular DiLoCo, but distributed almost uniformly over time instead of concentrated in a single instant.
In MaxText, this fragmentation follows three engineering decisions:
- P = number of layers + 1: one fragment for everything that isn't a layer (token embeddings, final layer norm, output head) and one fragment per Transformer layer.
- Staggered streaming: instead of pausing every H steps to exchange the entire model, MaxText synchronizes one fragment every
Δh = max(1, round(H/P))steps. With H = P, this becomes one fragment per step. - Embedding balancing: since the fragment for non-scanned layers carries the embedding matrices (which can be huge because of vocabulary size), MaxText distributes pieces of these large matrices across the other fragments, so that no single fragment becomes an isolated traffic spike.
The synchronization order is also interleaved by network depth (fragment 0, then layer k, then layer 2k, and so on) instead of sequential, which avoids leaving an entire section of the model outdated while another has already been synchronized.
The numbers: how much network bandwidth this actually saves
Google illustrates the gain with a sizing example: training Qwen3-32B in bfloat16, with a step time of about 6.2 seconds and P = H = 65.
| Network regime | Traffic per step | Minimum bandwidth required |
|---|---|---|
| Data Parallel | ~65.5 GB | ~85 Gbps |
| Vanilla DiLoCo (spike every 65 steps) | ~65.5 GB (all at once) | stalls on 10 Gbps links |
| Streaming DiLoCo | ~1.0 GB | ~1.3 Gbps |
The actual benchmark used two TPU v5p-32 slices and limited the bandwidth between them with Linux's tc, simulating three congestion regimes. The results match the theoretical calculation:
| Regime | Data Parallel | Vanilla DiLoCo | Streaming DiLoCo |
|---|---|---|---|
| No bandwidth limit | ~5.6-5.9 s/step | ~5.6-5.9 s/step | ~5.6-5.9 s/step |
| 30 Gbit/s aggregate | 19.41 s/step | 6.39 s/step | 5.76 s/step |
| 10 Gbit/s aggregate | 55.11 s/step | 7.55 s/step | 5.76 s/step |
In the most congested regime, Streaming DiLoCo delivers a 9.57x speedup over traditional Data Parallel and still outperforms vanilla DiLoCo, which continues to suffer from the periodic full synchronization pause every H steps. Total traffic measured on the DCN dropped from 4,258.8 GB to 67.0 GB over 65 steps, a reduction of 64 to 65 times.

The obvious question: does this make the model worse?
Letting learners train independently for dozens of steps before synchronizing raises a legitimate concern, especially in Mixture-of-Experts architectures, where the expert routers could diverge from each other. Google tested Streaming DiLoCo against synchronous Data Parallel on two MoE architectures, Qwen3-30B-A3B and Gemma4-26B-A4B, over 4,000 training steps.
The result: after an initial adjustment in the first ~800 steps, Streaming DiLoCo's loss curve tracks, and according to the post, even slightly outperforms, that of Data Parallel. Two implementation details explain this stability: the outer optimizer uses Nesterov momentum (β = 0.9) instead of simply averaging the weights, and synchronization updates only the model parameters, preserving the first- and second-moment buffers of each learner's local AdamW, which never loses its own optimization history.
How to enable it in MaxText
Configuration is done via YAML or command-line flags:
# Enables DiLoCo and Streaming DiLoCo
enable_diloco: true
enable_streaming_diloco: true
# Partitions into N_layers + 1 fragments (e.g., 64 + 1 = 65 for Qwen3-32B)
num_diloco_fragments: 65
diloco_sync_period: 65
use_sequential_layers: false # false = layers interleaved across fragments
# Outer optimizer hyperparameters (SGD with Nesterov)
diloco_outer_lr: 0.7
diloco_outer_momentum: 0.9
# Topology: number of learners (slices) spread across the DCN/WAN
dcn_diloco_parallelism: 2Under the hood, the diloco.py module splits the global batch among the M learners, each processing GBS / M tokens per step. To overlap communication and computation, the deltas for the scheduled fragment are offloaded to synchronization workers that run on the TPU hosts' own CPUs, sharded across M parts. These workers perform the fragment's all-reduce in the background while the TPUs keep running the inner loop steps, without relying on a central coordinator.
Who this matters for, and who it doesn't
No one is going to run this on a personal laptop: the technique assumes the team already has multiple TPU slices (or, in theory, GPU slices) across different datacenters or regions and is willing to deal with the complexity of orchestrating geographically distributed training. For those already facing this scenario, today forced to concentrate everything in a single expensive, scarce datacenter, Streaming DiLoCo changes the equation: instead of needing ~85 Gbps dedicated between sites to train a model the size of Qwen3-32B, ~1.3 Gbps is enough, an order of magnitude compatible with common WAN links between cloud regions.
The practical value for those who don't train models with tens of billions of parameters is more conceptual: the roofline model used in the post (required bandwidth ≈ data volume divided by the computation time to overlap) and the logic of fragmenting what needs to be synchronized instead of synchronizing everything at once are patterns that show up in any distributed system with a network bottleneck, not just AI training. It's worth studying the code even without any intention of running a TPU cluster.
Google says Streaming DiLoCo is available in MaxText starting with version 0.2.5, published on PyPI, with corresponding Docker images and code on the project's main GitHub branch. The post also references a theoretical link-sizing guide, a step-by-step tutorial with recipes for Qwen3 8B and 30B, and ready-made launch scripts for the dense and MoE scenarios.
Translated from the Brazilian Portuguese original · Read the original
PostgreSQL 19 arrives with WAIT FOR LSN to solve the problem of outdated replicas
Released in November 2026 after a two-month delay caused by a wave of AI-assisted vulnerability reports, PostgreSQL 19 brings the WAIT FOR LSN command, which lets an application guarantee that it has just read, on a replica, the data it itself wrote to the primary database.





