How to Debug AI Agent Failures in Production with Session Traces and Cost Limits

An AI agent can call the wrong tool repeatedly without triggering a single availability alert. The service responds, the uptime metric stays green, and yet the autonomous workflow is burning tokens in a loop, hitting invalid endpoints, or claiming to have completed a task it actually skipped. That's the scenario Sabith K Soopy, principal engineer at StackGen, describes in a post published in the CNCF community on August 4, summarized by InfoQ.

The line that opens the piece gets straight to the point for anyone running agents in production:

The hardest part isn't building them; it's understanding what they're doing when they go wrong.

>

-- Sabith K Soopy, principal engineer at StackGen

The structural problem is familiar to anyone who's already hooked up APM to a traditional service: classic monitoring tells you whether a service is up, not why an autonomous flow got stuck in a loop, delegated to the wrong sub-agent, or hallucinated a tool call. For the Brazilian developer rolling out internal copilots, support automations, or RAG pipelines, this matters because the LLM API bill arrives at the end of the month, and a silently looping agent is money evaporating with no red flag on the dashboard.

Session Traces: Every Call Becomes a Span

The foundation of the approach described is capturing nested session traces. StackGen uses Langfuse to log every LLM call, every tool execution, and every sub-agent delegation as an individual span, with execution latency and token cost attached.

The detail that makes a difference in multi-agent workflows is nesting: placing child spans beneath parent traces preserves the full delegation chain. Without it, you're left with a pile of loose calls and no way to reconstruct which agent asked what of whom. It's the difference between a log of isolated lines and a readable distributed trace.

One operational point the post emphasizes: the span exporter should be asynchronous and batched, queuing spans in memory and flushing periodically. That way, if the telemetry backend goes down temporarily, you lose trace data, but the agents keep running. The opposite (blocking, synchronous export) would turn an observability outage into an application outage, something nobody wants.

Cost Controls Before Execution, Not After

Here's the part that most changes the bill at the end of the month. The piece treats cost controls as the primary operational safeguard against runaway execution, and the recommendation is to act before execution starts:

• Hard iteration caps and per-tool-call limits, set before the agent runs;
• Pre-execution checks that block identical, consecutive requests to the same tool.

Blocking identical consecutive calls solves the dumb repetition, the obvious loop. But the post argues that isn't enough and proposes combining it with statistical monitoring: comparing each session's cost against that agent's moving average to flag subtler anomalies, including model routing errors, tool hallucinations, and runaway context expansion across multi-turn interactions.

The reasoning behind this matters for anyone designing the system: reactive alerts arrive too late for fast, parallel agents. By the time the alert fires, the agent has already run through dozens of iterations. That's why the brake needs to sit in the execution path (the caps), not just on the alerting dashboard.

Traces Are for Debugging, Metrics Are for Alerting

One of the post's most concrete lessons is about a classic instrumentation mistake: stuffing dynamic identifiers into metric labels.

Traces are for debugging, metrics are for alerting.

>

-- Sabith K Soopy, principal engineer at StackGen

The recommendation is to export only bounded operational metrics to Prometheus (tool error rate, approval latency histograms, for example). Putting dynamic session IDs as metric labels creates high-cardinality time series, which can literally take down the metrics server. Every new ID spawns a new series; in production with lots of sessions, it explodes.

The granular context of each session belongs strictly to traces or structured logs, not to metrics. It's a rule that applies to any system, but AI agents, with their volume of calls per session, make it even more dangerous to ignore.

Append-Only Logs and a Diagnostic Tool

For post-incident analysis, the post recommends recording tool calls, governance decisions, and memory operations in an append-only, searchable log, with credentials and personal data (PII) redacted before storage. Immutable logging matters here because, during incident investigation, you need to trust that the record wasn't altered after the fact.

StackGen complements this with a command-line diagnostic tool that validates, in a single run:

• model API access;
• vector database reachability;
• pending approvals;
• memory item count;
• connection to the trace backend;
• integration health.

It's the kind of comprehensive healthcheck that saves time when something breaks and you need to quickly know which link in the chain failed. On top of that, complete traces run through automatic analyzers that flag execution duration, tool failures, retry counts, and token efficiency issues for human review.

The Ecosystem Around It: OpenTelemetry, LangSmith, Phoenix

The post situates these practices within a set of tools that's already consolidating, and that's worth knowing before locking in your stack:

| Tool | Role | |---|---| | Langfuse | Captures nested session traces with cost and latency per span | | OpenTelemetry GenAI | Semantic conventions that standardize attributes for model operations, token consumption, and tool invocation | | LangSmith | Converts anomalous production traces into test datasets for regression benchmarking | | Arize Phoenix | Open source, native OpenTelemetry tracing with self-hosted evaluation via LLM-as-a-judge and prompt experimentation | | Prometheus | Alerts on bounded operational metrics (never session IDs) |

The point about OpenTelemetry's GenAI semantic conventions deserves attention from anyone looking to avoid lock-in: they establish a consistent attribute schema across different telemetry backends. The project maintains these specifications in a dedicated repository that covers client, server, and Model Context Protocol (MCP) spans, which helps keep observability consistent across multiple vendors.

There's a conceptual distinction the piece makes clear that rounds out the picture: a trace records the execution history; evaluation tools check the quality of the output. These are different things. Knowing the agent ran isn't the same as knowing it ran well. LangSmith and Phoenix operate at that second layer, taking real production traces and turning them into a base for testing quality and regression.

What's Left Open

The piece is an account of operational practices from one company, StackGen, based on months of running agents in production, not a benchmark with published cost-reduction numbers. There are no disclosed metrics on how much the iteration caps cut from the bill, or how many incidents the append-only log helped resolve. The useful takeaway for the Brazilian developer is the map: where to instrument (nested-span traces), where to brake (pre-execution caps), where to alert (bounded metrics in Prometheus), and where not to put high cardinality. Testing these pieces in your own agent pipeline, measuring cost before and after, is the step left for whoever puts it into practice.