Google Brings Pathways, Gemini's Training Architecture, to General Availability on Cloud

Pathways replaces the rigid synchronization of DDP and FSDP with a single client that describes the entire training run as a graph, using asynchronous dispatch and a central scheduler to handle ordering between blocks.

Google announced this Monday (November 9, 2026), in a post on the Google Developers Blog, the general availability (GA) of Pathways on Google Cloud. It is the distributed system that Google itself uses internally to train its largest models, including Gemini, and that any team can now contract as a service.

Pathways' proposal is not to compete with DDP or FSDP in the same category: it is to change the execution layer beneath them. While DDP and FSDP follow the SPMD (Single Program, Multiple Data) model, in which every accelerator runs the same copy of the script and synchronizes via collectives like AllReduce, Pathways introduces a single Python client that describes the entire computation as a graph, letting the runtime decide where each part runs, how tensors move, and in what order everything happens.

Where SPMD Gets Stuck

SPMD works well when the job is a single program, running identically on every chip. The problem appears when the work stops being uniform. Pipeline parallelism, for example, splits the model into blocks that run different code on different devices: while one block processes the first micro-batch, the others sit idle, incurring cost without producing anything (the post calls this a "pipeline bubble").

Another case cited by Google is reinforcement learning (RL) post-training for LLMs: a sampler generates responses, a reward model evaluates them, a trainer updates the policy and returns new weights to the sampler. These are three distinct programs that need to overlap; if they run sequentially, the sampler sits idle while the trainer works, and vice versa.

In both scenarios, no individual process sees the entire dependency graph, because each worker only knows its own devices and data. This has three practical consequences, according to the post:

• A failure brings down the entire job. Because each process holds its accelerators from start to finish and every collective blocks until all finish, losing one machine interrupts the whole collective and forces a restart from the last checkpoint. Google notes that this interrupted Gemini 2.5 training multiple times per hour.
• Capacity cannot be reallocated. A stage that needs fewer chips keeps holding onto the surplus ones until the job finishes.
• Debugging across multiple hosts is painful. There is no single process where you can set a breakpoint; when it hangs, each worker sits waiting on the collective, with no central place showing the entire job.

How Pathways Solves This

With Pathways, the developer writes against virtual devices, not physical ones. A resource manager maps these virtual devices to real chips, and a worker on each accelerator host executes the compiled programs it receives. This indirection is what makes it possible to remap work to healthy chips when a host goes down, instead of relaunching the entire job, which is the basis for what Google calls elastic training.

The same resource manager allows a single Pathways cluster to serve multiple clients at the same time. In the Shared Pathways Service, the cluster runs as a long-lived service, and each client connects on demand (from a laptop or VM), receives its TPU slices from the shared pool, and returns them upon disconnecting.

The difference in network scale explains why this matters. On a TPU v5p, each chip has 1,200 GB/s of bidirectional bandwidth within its tightly connected group, but only about 6 GB/s (50 Gb/s) for the datacenter network between groups, two orders of magnitude less. Since each SPMD step blocks on its collectives, the more groups a job spans, the larger the share of each step spent moving data instead of computing.

Asynchronous Dispatch and Central Scheduler

To keep accelerators busy, Pathways dispatches using futures and parallel asynchronous dispatch. While block A is still computing, the runtime already knows the input shapes and dtypes for block B, so it allocates B's buffers and schedules the transfer before A's output even exists. This keeps the single client off the critical path: coordination happens in parallel with what is already running on the chips, instead of blocking the accelerator between steps.

The second mechanism is a central scheduler. In pipeline training, the devices of a block often have both the forward pass of a new micro-batch and the backward pass of an older one ready to run at the same time. If each device decided on its own which to execute first, the collectives would conflict and the group would hang. Pathways solves this by placing the operations of all programs in a single consistent order, ensuring that operations that need to communicate start together.

Pathways Is Not Ray

Google's post anticipates the obvious comparison with Ray, the open-source framework for orchestrating Python workloads on a cluster. The difference is one of layer: Ray provisions the cluster, schedules Python code as tasks and actors, and treats the accelerator as a resource to allocate, but it does not change how training runs inside each worker, which remains pure SPMD, with the usual collectives.

Pathways operates exactly at the layer Ray doesn't touch: execution within the step. The two are, according to the post itself, complementary, not competing. In theory, you can use Ray to orchestrate the data, training, and serving pipeline, and Pathways to control how each training step executes within the chips reserved by Ray.

What Changes for Those Training Models in Brazil

The most important caveat for anyone reading this with an eye toward applying it tomorrow: Pathways runs JAX programs on TPU, it is not a direct substitute for DDP/FSDP for those who have already invested in PyTorch on GPU. This restricts adoption to teams that are already on (or considering migrating to) JAX and TPU infrastructure on Google Cloud, a scenario still in the minority outside big tech companies and research labs that publish in JAX.

That said, the practical value of the announcement isn't only for those who will literally use Pathways. The diagnosis the post makes of SPMD, blocking collectives, a failure that takes down the entire cluster, the difficulty of distributed debugging, is the same bottleneck that any team running FSDP across multiple GPU nodes faces today. Understanding that an architectural alternative exists (single-controller with an asynchronous dataflow graph) helps evaluate, for example, whether it's worth investing in JAX/TPU ecosystem solutions for workloads that combine training and RL post-training, where the pipeline bubble and the wait between sampler, reward model, and trainer cost dearly on GPU rented by the hour.

Anyone choosing between MaxText (one of the repositories Google lists as a resource complementary to Pathways) and a traditional PyTorch stack now gains a third criterion beyond chip cost and team familiarity: tolerance for hardware failure in long runs. For training runs of days or weeks on large clusters, the promise of remapping around a dead host without restarting from scratch truly matters; for short fine-tuning on a handful of GPUs, the gain probably doesn't offset the learning curve of migrating from PyTorch to JAX.

Along with the announcement, Google published the Pathways on Google Cloud documentation, the academic paper "Pathways: Asynchronous Distributed Dataflow for ML," the resilient training documentation, and the pathways-utils and MaxText repositories on GitHub, enough material for anyone already running JAX to evaluate a proof of concept before deciding whether to migrate an entire training workload.