Google shows how the Agents CLI catches AI agent regression before production

The problem that lint and unit tests don't catch

An agent that works in the demo and an agent that keeps working after model swaps, tool changes, and new conversation formats are two different problems. The first takes a weekend. The second takes a loop: write test cases, run them, score them, read what failed, change the agent, run again. If the score goes up, the change was good. If it drops, revert.

Pier Paolo Ippolito and Elia Secchi, from Google, published a complete walkthrough of this cycle on the Google Developers Blog on October 28, 2026, using the Agents CLI's eval namespace. The technical name for this process of gradually raising the score is hill-climbing, and the value of the piece lies in the real numbers it shows at every step, including when a change makes the agent worse.

The five commands of the eval namespace

The Agents CLI's eval is small and framework-agnostic:

• eval generate: runs the agent against a dataset of cases and records the traces
• eval grade: scores the generated traces using the configured metrics
• eval compare: produces a structured diff between two results
• eval analyze: groups failures into themes
• eval optimize: runs an automated hill-climbing pass on the prompt

Setting up the project and the dataset

The scaffold used in the walkthrough is the LangChain template that ships with Agents CLI v1.8.0:

agents-cli create my-agent \
--agent google/agents-cli/extensions/langchain/template@v1.8.0 \
-d cloud_run
cd my-agent && agents-cli install

The project already comes with a dataset at tests/eval/datasets/basic-dataset.json, a config at tests/eval/eval_config.yaml, and an LLM-as-judge metric at tests/eval/response_quality.py, which scores each response from 1 to 5 on accuracy, relevance, and clarity using gemini-3.8-flash as the judge. The default dataset has a single trivial case, insufficient for a real hill-climb.

The team expanded it to four: a greeting, a weather question that the mock tool answers with a fixed value, a weather question for another city, and an out-of-scope question (who won the 2024 World Series). The dataset is small on purpose: the loop only pays off when it runs on every change, not when it becomes an hour-long release gate.

The first run (baseline)

With the dataset ready, eval generate runs the agent against the four cases and eval grade scores the resulting traces. The result of the first run: a mean of 4.0 out of 5, standard deviation 1.4142. That deviation matters: a uniform 5.0 score across every case would mean either a perfect agent (unlikely) or a dataset too easy to show variation. A 4.0 with real spread is a hill worth climbing.

Injecting a regression on purpose

To demonstrate the loop, the team simulated a common mistake: a developer working on several agents pastes the wrong system prompt, from a recipe project, into the weather agent.

# Before
system_prompt="You are a helpful assistant"

# After
system_prompt=(
 "You are a professional chef. Help users with recipes, cooking "
 "techniques, and ingredient substitutions. Always suggest a recipe "
 "at the end of every response."
)

The code compiles, the agent keeps running, and nothing in the lint, unit test, or type-check pipeline catches the problem. Running eval generate and eval grade again: a mean of 2.75, standard deviation 0.9574. A drop of 1.25 points on a 1-to-5 scale is the kind of regression that slips past a code review where the diff looks small ("just one string changed").

Where the regression hit, case by case

The mean alone signals that something got worse; eval compare shows where. The command reads two result files and returns a structured JSON diff with the keys that changed and the case-by-case comparison.

The most serious case is sf_weather: the chef persona pulled the model's attention away from the registered get_weather tool, and the agent stopped calling it. No unit test catches this, because the code runs and the tool stays registered; only an end-to-end evaluation that judges the response in context reveals that the model's behavior changed.

Switching models is cheaper, but is it worth it?

A common hill-climbing question is whether a cheaper model delivers acceptable quality. The walkthrough swapped gemini-3.8-flash for gemini-2.5-flash (restoring the correct system prompt first) and compared the results.

The mean dropped from 4.0 to 2.75, the same magnitude as the chef regression, but for a completely different reason: the cheaper model tied on two cases and only collapsed on off_topic, refusing to answer with a caveat that the event "hadn't happened yet" (an artifact of its training cutoff), which the judge penalized heavily. Looking at the mean alone would lead to dismissing gemini-2.5-flash as if it were as bad as the broken prompt; looking at the per-case detail shows a reasonable trade-off for most of the workload, with a specific limitation on out-of-scope questions about recent events.

Two caveats here, according to Google's own text: a prompt tuned for a stronger model tends to leak capability when ported to a weaker one, so the correct comparison is "weak model with its own prompt" against "strong model with its own prompt." And the judge model (which produces the score) needs to stay constant between the two runs; swapping the judge is a different experiment.

Scheduling the loop and triggering an alert

The Agents CLI doesn't come with a built-in scheduler, but the eval commands are built to plug into standard infrastructure: a cron job or Cloud Scheduler running eval generate and eval grade every day against a larger regression dataset (dozens to hundreds of cases, since it runs only once a day), saving the result in a dated folder.

A result file that nobody looks at isn't an alert. The suggested pattern is a script that compares the mean of the new run against a baseline and exits with an error code when the drop exceeds a threshold:

b, c = mean(args.baseline), mean(args.candidate)
drop = b - c
if drop > args.threshold:
 print(f"FAIL: mean dropped {b:.3f} → {c:.3f}")
 sys.exit(1)
print(f"OK: mean {b:.3f} → {c:.3f}")

Run against the baseline 4.0 / candidate 2.75 pair from the chef regression, with a threshold of 0.1, the script prints FAIL: mean dropped 4.000 → 2.750 and exits with code 1; against two identical results, it exits with OK and code 0. It's this exit code that a GitHub Actions, cron, or Cloud Scheduler job uses to trigger the alert.

Automated prompt optimization with GEPA

For ADK projects (not for the LangChain extension used in the rest of the walkthrough), eval optimize runs an automated hill-climbing pass using the GEPA framework: it proposes prompt variants, scores each one, keeps the ones that improve the metric, and iterates. The command is scoped only to the prompt, not to the agent's tools, structure, or model.

Google itself warns that GEPA sometimes finds genuine improvements and sometimes finds variants that score higher by overfitting to the specific metric. The recommended discipline is to run eval optimize on the regression dataset, where the improvement is measurable, and review each candidate before promoting it into the agent.

What changes for those building agents here

One data point from the piece deserves special attention: a controlled study by Google itself found that coding agents using the Agents CLI run four times more evaluations than agents without it, but change the agent after an evaluation at the same rate as the baseline. In other words, the behavioral gain comes from what goes into the build from the start, not from the volume of internal iteration.

In short: evaluation is necessary but not sufficient; quality comes from acting on what the evaluation shows, not from running it more times. For those building agents in Brazil, the walkthrough's practical lessons matter more than the announcement itself: a small dataset that runs on every commit beats a giant dataset that becomes shelfware, the full result file is the evidence (not just the mean), the dataset and the expectation config should live in separate versioned files, and the mean alone lies, as the model swap showed.

Those who want to reproduce the loop should start with uvx google-agents-cli setup and agents-cli create my-agent; the complete documentation for the eval namespace is at google.github.io/agents-cli/cli and the source code is at github.com/google/agents-cli.