AIARTICLE

Tunix and Gemma 4 show how to train multi-turn AI agents on TPUs

Tunix and Gemma 4 show how to train multi-turn AI agents on TPUs
Image: Alan Andrade
Article · AI
Reading
6 min
Tables
1
Code
3
In short

A Google Developers Blog tutorial published on November 19 uses the Plane Strike game to teach, step by step, how to train an agent that calls tools and corrects its own plan across multiple turns.

The problem: real agents don't finish the job in a single response

In two previous posts, the Google Developers Blog showed how Tunix, Google's JAX library for LLM post-training, simplifies single-turn supervised fine-tuning (SFT) and GRPO (Group Relative Policy Optimization) on Cloud TPUs. But a real agent rarely solves a problem in one shot: it needs to observe a partial result, call a tool, read its response, and adjust its strategy across multiple turns before finishing the task.

On November 19, 2026, Wei Wei, a developer advocate at Google, published a complete tutorial showing how to train a tool-using multi-turn agent with Tunix's agentic reinforcement learning pipeline, running on a single TPU v6e-8. The entire code is in the planestrike-gemma4-rl repository on GitHub.

Battleship as a test bed

To isolate the multi-turn reasoning problem without the complexity of a real-world environment, the team reused the Plane Strike game from a previous post on RL in TensorFlow/JAX, replacing the custom MLP policy with Gemma 4 E2B in thinking mode.

The rules: a 6x6 board (A1 to F6), an 8-cell plane hidden in one of 4 possible orientations. On each turn the agent sees the result of the previous move and the visible board (. untried, X hit, O miss), reasons about where to attack, and calls the strike(cell="...") tool. The game ends when all 8 cells of the plane are hit or after 23 attempts.

Untrained, google/gemma-4-E2B-it already has some spatial awareness and knows how to call tools, but it only wins 9.4% of games on evaluation boards, taking an average of 22.44 attempts. After 100 steps of agentic GRPO with Tunix, the win rate rises to 95% and the average game drops to about 16 attempts.

The win rate chart (win_rate) rises from approximately 0.2 to nearly 1.0 over the GRPO training steps in the Plane Strike game
The win rate chart (win_rate) rises from approximately 0.2 to nearly 1.0 over the GRPO training steps in the Plane Strike game. Reproduction: developers.googleblog.com.
MetricBefore trainingAfter 100 steps
Win rate (evaluation)9.4%~95%
Average attempts per game22.44~16

Four modular pieces make up the pipeline

The tutorial organizes the training into four components, each in its own file:

  • Task environment: PlaneStrikeEnv, a subclass of BaseTaskEnv, in env.py
  • Tool and chat template parser: BaseTool and Gemma4ChatTemplateParser, in strike_tool.py
  • Conversation agent: ConversationAgentBase, in agent.py
  • Trajectory collection and the GRPO learner: TrajectoryCollectEngine and GRPOLearner, in trajectory_utils.py and train_planestrike.py

Each environment instance receives a dictionary with a seed that fixes the hidden plane's position, ensuring that the 8 rollouts in the same GRPO group play exactly the same board:

python
class PlaneStrikeEnv(BaseTaskEnv):
  """One Plane Strike game. `entry["seed"]` fixes the hidden plane position."""

  def __init__(
      self,
      entry: dict[str, Any],
      group_id: int | None = None,
      pair_index: int | None = None,
      max_steps: int = MAX_STEPS,
      **kwargs,
  ):
    entry = {k: _unwrap(v) for k, v in entry.items()}
    self.seed = int(entry.get("seed", 0))
    self.split = str(entry.get("split", "train"))
    super().__init__(task=dict(entry), max_steps=max_steps, **kwargs)
    self.extra_kwargs["group_id"] = group_id
    self.extra_kwargs["pair_index"] = pair_index
    self._reset_board()

At each step, _step_impl evaluates the attacked cell against the hidden board, updates the visible grid, adds the turn's reward to the whole trajectory's reward, and returns an EnvStepResult(observation, reward, done, info). For agents with multiple standard tools, the tunix.rl.agentic module already provides ready-made ToolEnvironment, ToolAgent, and ToolManager: this custom-built code is only needed when turn control is special.

How Gemma 4 calls tools without leaving its own turn

The most specific point of the tutorial is how Gemma 4 handles tool calls natively, without an external function-calling framework. The tool schema is declared in the system prompt inside <|tool>declaration:...<tool|>, and a model turn that calls a tool ends with <|channel>thought ... <tool_response|>. That's one of Gemma 4's native stop tokens: the model pauses, receives the response as response:strike{value:"..."}<tool_response|>, and continues generating its own reasoning and the next call within the same assistant turn.

For this to work, the post extends Tunix's default parser:

python
class Gemma4ToolChatTemplateParser(parser_lib.Gemma4ChatTemplateParser):
  """Gemma4ChatTemplateParser plus native tool declarations and in-turn tool
  responses."""

  def __init__(self, tokenizer, tools, enable_thinking: bool = True):
    super().__init__(tokenizer, enable_thinking=enable_thinking)
    self.tool_declarations = render_tool_declarations(tokenizer, tools)
    self.tool_response_id = tokenizer.convert_tokens_to_ids(TOOL_RESPONSE_START)

  def _parse_tool(self, content: str) -> str:
    return f"response:{TOOL_NAME}{{value:{_STR}{content}{_STR}}}<tool_response|>"

  def parse(self, messages, add_generation_prompt=False, is_first_msg=False):
    if messages and messages[-1]["role"] == "tool":
      add_generation_prompt = False  # The model continues its own turn.
    return super().parse(messages, add_generation_prompt=add_generation_prompt,
                          is_first_msg=is_first_msg)

This matters because it avoids retokenizing the entire conversation on every tool call, one of the most common ways to introduce noise between what the model generated and what training sees as tokens.

The agent and the asynchronous GRPO loop

In agent.py, PlaneStrikeAgent bridges environment observations and model messages. It ignores everything inside <|channel>thought...<channel|> when validating a move, and only counts as a valid action a final response with exactly one well-formed strike call; everything else becomes INVALID_ACTION.

Training runs with Tunix's asynchronous vllm-tpu engine, on a TPU v6e-8 where the actor model, the reference model, and the rollout workers share the 8 chips. The GRPO configuration uses loss_algo="gspo-token", advantage_estimator="rloo", and exact_token_continuity=True to preserve exact token IDs across turns, without retokenization. The 100-step job is launched with:

shell
python train_planestrike.py \
  --run_dir runs/grpo_6x6_tool_thinking_100 \
  --learning_rate 2e-6 \
  --lr_end 2e-7

What the numbers show throughout training

Three curves tell the story of the training, according to the post:

  • Tool-call compliance converges first: tool_call_rate approaches 100%, invalid drops close to zero, and repeats (repeated attacks on the same cell) falls from ~0.55 to ~0.15.
  • The win rate rises from ~0.20 in the first training rollouts (9.4% in held-out evaluation) to ~0.95.
  • The average game length drops from ~22 to ~16 attempts.

The post itself acknowledges a limitation: today the entire game uses a single trajectory reward, summed turn by turn. Further gains, according to the authors, depend on finer per-turn credit assignment, that is, rewarding each move individually instead of only the final outcome.

What changes for those building agents here

In short: the tutorial isn't about playing Plane Strike, it's a reproducible recipe for multi-turn RL with native tools, without relying on a function-calling orchestrator outside the model. This matters to anyone building agents that need to maintain state across calls, such as a support assistant that queries a system, or a data agent that runs a query, refines it, and runs it again.

It's worth noting the practical constraints before replicating this in a project in Brazil:

  1. The entire pipeline is designed for Cloud TPU v6e-8; running on GPU requires adapting the vllm-tpu rollout and Tunix's cluster configuration, which is not trivial.
  2. tunix.rl.agentic with ToolEnvironment, ToolAgent, and ToolManager covers the common case of multiple tools; it's only worth writing an environment and parser from scratch, as in this tutorial, when turn control requires something outside the standard pattern (here, Gemma 4 continuing its own turn after the tool's response).
  3. The jump from 9.4% to 95% win rate came from a deterministic game that's cheap to simulate; the same learning curve isn't guaranteed in tasks where the environment (a real API, a database) is more expensive or slower to run at scale.

Tunix is open source, and the planestrike-gemma4-rl repository provides the complete code, including the env.py, strike_tool.py, agent.py, and train_planestrike.py files cited in the post. For those already using single-turn GRPO with Tunix, the leap to multi-turn here is more about reusing the library's ready-made agentic abstractions than rewriting the training from scratch.

Translated from the Brazilian Portuguese original · Read the original

View profile →