Google publishes codesign guide to extract the most from TPUs in AI models

- Reading
- 8 min
- Tables
- 1
- Code
- 1
Google released the tpu-codesign guide on Monday (11/16), with eight architecture principles to align LLMs and MoE models with the hardware of Ironwood TPUs and upcoming generations, with measured gains of up to 57% in training throughput.
What Google published
The Google Developers Blog published the open repository AI-Hypercomputer/tpu-codesign on Monday (November 16, 2026), authored by Gagik Amirkhanyan and Vaibhav Singh, engineers on the TPU team. The material is not just another hardware announcement: it is a codesign guide, that is, about how to design the model architecture (layer dimensions, MoE routing, parallelism strategy) already thinking about the chip's physical behavior, before writing the first training step.

This matters because the text acknowledges something every team that has tried to port a model from GPU to TPU feels firsthand: a tensor shape decision made at model definition time becomes a permanent ceiling on MFU (Model FLOP Utilization), cost, and serving latency. It can't be compensated for later with more parallelism or more chips.
Why TPU architecture is different
TPUs trade a GPU's generic scaling for a much more rigid design: wide 256×256 systolic arrays (the MXU), dedicated SparseCore engines for sparse operations and MoE routing, and a fixed multidimensional interconnect fabric (ICI). This specialization is why TPUs deliver more performance per total cost of ownership (TCO), but the price is that they punish any shape misalignment or access pattern that a GPU would simply absorb.
The guide organizes the recommendations into three blocks, in order of risk:
- Shape rules (principles 1-2): dimension alignment and static shapes, virtually zero quality risk, always adopt;
- Architecture choices (principles 3-7): structural changes that generally carry a quality trade-off and require ablation before going to production;
- Parallelism expression (principle 8): compiler and kernel tuning that doesn't require touching the checkpoint.
If the team is designing a new model, the shape rules always apply. If it is adapting an already trained model, only the parallelism section applies without reopening the checkpoint.
Shape rules: aligning to 256 without a quality cost
The simplest recommendation to apply, and the one the guide itself calls the most frequently ignored, is to make d_model, d_ff, latent widths, and padded vocabulary multiples of 256, matching the systolic MXU size. The most common mistake is head_dim: at 64 or 128, flash-attention's QK products leave the MXU idle 50% or more of the time.
The fix reduces heads and increases head_dim, preserving the total parameter count and KV cache size:
# before: 32 heads x 128 head_dim
num_heads, head_dim = 32, 128
# after: 16 heads x 256, same parameters and total FLOPs
num_heads, head_dim = 16, 256In Google's internal profiling with a 30B MoE derived from Qwen3-30B, this single change accelerated end-to-end training, with growing gains at longer context lengths. The same pattern shows up in ablations of non-quadratic attention (Gated DeltaNet), reinforcing that this isn't a quirk of just one architecture.
The second rule deals with dynamic shapes, common in MoE with dropless routing and in paged attention. Instead of padding buffers to the largest group or the longest sequence, the guide recommends keeping shapes fixed and describing the irregular structure with small integer arrays (group sizes, offsets, page table), copied into the TensorCore's scalar memory via Pallas's PrefetchScalarGridSpec before the grid runs. MaxText already exposes this as packing=True: the data dependency becomes an address and loop counter, not a shape change, and the compiled program runs the same regardless of batch composition.
LatentMoE: compressing routed token traffic
The third principle is the guide's most structural one. In standard MoE, each routed token travels at the full d_model width between dispatch and combine, which, according to Google, makes combining expert parallelism (EP) with FSDP slower than pure FSDP in Ironwood benchmarks. The proposed solution, called LatentMoE (a design already used by Nemotron 3, MAI-Base-1, and Kimi K3), projects the token into a latent width ℓ that is 2 to 4 times smaller before dispatch, runs the experts in this compressed space, and only expands back after combine.
The golden rule for not losing quality in the trade is to keep K_new × ℓ ≤ K_old × d_model, reinvesting the saved bandwidth into more active experts per token. In Google's profiling of the 30B / 128-expert MoE, an ℓ = 768 (a 2.67× compression) cut on-device permutation time by 2.8× and raised training throughput by 14% (from 270 to 322 TFLOP/s per chip), besides freeing up about 25 GB of HBM per device when recompute checkpoints were anchored in the latent space.
The trade-off is worthwhile when the per-device batch is too small to hide FSDP's weight all-gathers: this is exactly the scenario where EP with LatentMoE and FP8 dispatch becomes competitive. The fourth principle complements this idea: between two MoE layouts with equivalent quality, prefer the one that concentrates more work in dense, communication-free GEMMs (a wider shared expert) instead of multiplying small experts.
In a 63B MoE with ℓ = 768 and a K/N = 1/32 ratio, reducing from (512, 16) to (256, 8) active experts, doubling d_ff and the shared expert, raised throughput by 57% while maintaining a lower loss over 120.8 billion tokens.
The decode bandwidth wall
Serving a large model at low concurrency means reading the model's entire weights for every generated token. The guide estimates that, in dense FP8, something like 300 concurrent tokens per weight shard are needed to hit the compute ceiling. Three levers address this: low-diameter topologies, speculative decoding via multi-token prediction (MTP), and explicit concurrency sizing.
In practice, this means calculating the latency floor as the ratio between weight bytes per chip and HBM bandwidth (the example cited is Qwen3.5-397B in FP8 streaming ~100 GB per chip across 4 chips, a floor of ~13.5 ms before even touching the KV cache), and sizing the maximum batch by the remaining HBM divided by the KV cost per sequence. For MoE, weight traffic scales with the union of experts touched in the batch: if real concurrency will be low, Google's recommendation is to design fewer active parameters, not hope for a batch that won't exist.
VPU, precision, and where FP8 doesn't pay off
The sixth principle tackles a less obvious bottleneck: norms, activations, masks, and quantization scales run on the VPU (Vector Processing Unit), about two orders of magnitude slower than the MXU, and execute serially with the matmuls unless XLA manages to fuse this work into a neighboring GEMM. The recommendation is to standardize on fusible RMSNorm and SwiGLU and to offload MoE's sort/gather/scatter to the SparseCore.
The guide details the math of FP8 with block scaling: with blocks of size B, the VPU rescale only stays "hidden" within the MXU window when B ≥ 1.5 × r_mxu/r_vpu. On TPU v7x this ratio is close to 240, which pushes the practical threshold to blocks of about 340 elements: block 512 runs in the time of the whole tensor, while block 128 ends up 2.4× slower from being stuck on the VPU.
The seventh principle deals with the opposite case: for precision-sensitive matmuls, such as quadratic attention in long contexts, multi-pass BF16 only pays off when the throughput gain of the lower-precision format exceeds the number of passes: on Ironwood, the FP8/BF16 ratio is exactly 2:1, so no multi-pass scheme beats native BF16.
Mapping parallelism to physical links
The eighth principle is the only one that doesn't require reopening the checkpoint: mapping communication-heavy parallelism axes to the fastest available physical links. The practical rule for training is TP = 2 at most (mapped to the dual chiplets), reserving larger TP only for decode, which is limited by weight reads, not compute. FSDP remains the default, with EP restricted to a short axis of the 3D torus and DP/PP preferably pushed outside the pod, over the DCN network.
Enabling asynchronous collective offload on the SparseCore, according to tpu-recipes benchmarks, cuts 15% to 22% of step time with no model changes at all: it's worth measuring before ruling it out.
What changes in practice for those training outside Google's team
The table below summarizes where each TPU generation cited in the guide changes the calculation:
| Generation | Differentiator |
|---|---|
| TPU v7x (Ironwood) | the guide's baseline: 256×256 MXU, 4 SparseCores, 3D ICI torus |
| TPU 8t | native FP4, more balanced VPU for training |
| TPU 8i | native FP4, Boardfly topology, Collectives Acceleration Engine, 3x more SRAM |
| TPU 8i Max | doubles compute, SRAM, and HBM bandwidth per package |
| TPU 8tx | adopts the same Boardfly topology |
It's worth noting the warning Google itself makes: the "Measured" numbers come from the company's internal profiling program, with open 20-30B MoEs (a LatentMoE variant derived from Qwen3-30B and GPT-OSS-20B) running in BF16 against already-tuned baselines: in other words, these are transferable trends, not percentages that any team will reproduce exactly.
For those working with JAX and the MaxText stack on Google Cloud, the practical yardstick is clear: start with the shape rules (zero risk), run the LatentMoE ablation if the model is a MoE with more than 100 experts, and only then touch parallelism. For those on GPU, the shape alignment and LatentMoE principles still hold as architectural reading, but the physical topology part (torus, Boardfly, SparseCore) is TPU-specific and doesn't transfer without adaptation.
Translated from the Brazilian Portuguese original · Read the original
Cimed and Enter: when the founder's face becomes the brand's trust channel
At BRASA Summit Europe, Brunno Matias heard how Cimed and Enter use the voice of the people doing the work to build trust.





