AI That Claims to Have Completed a Task Can Now Be Found Through Search

AI applied to debugging got a new feature in LangSmith, announced by LangChain on October 14th. The feature is called AI Search. It lets you search through agent traces using natural-language descriptions of behavior.

The most-cited use case draws attention. It makes it possible to find situations where an agent incorrectly reports that it completed a task.

Other examples make the list. However, these are unwanted responses and exposure of personally identifiable information.

Why the error filter misses this case

Here's the problem the feature tackles. OpenTelemetry's log specifications recommend identifying failed operations through exceptions or reported errors.

Moreover, however, an operation can complete without reporting any error at all. This happens even when the agent failed to fulfill the user's request.

Consider the concrete scenario. However, the agent successfully invokes an external tool and claims to have completed the task, without actually carrying out the requested work.

Therefore, identifying this behavior requires examining the interaction history. Relying only on error attributes leaves the case invisible.

How the search works under the hood

The core component is called SmithDB. This database converts incoming traces into standardized trajectories.

The format brings together messages, tool calls, and results in chronological order. It also removes duplicate conversation histories.

Compatibility covers several frameworks. These include LangChain, LangGraph, OpenAI, Anthropic, Vercel's AI SDK, and OpenTelemetry.

The query flow happens in two stages. First, structured filters narrow down the set of candidate trajectories.

Then, a decision model examines the remaining records. It also assesses whether the described behavior actually occurred.

This approach differs from keyword search and vector search. According to the company, this helps distinguish an agent that completed the task from one that only claimed to have completed it.

Processing happens in parallel. Results arrive as they appear, without requiring a wait for the scan to finish.

AI in long sessions runs into the context budget

The company's own numbers reveal the scale of the challenge. Across 150 conversations from Open SWE, the company's open-source coding agent, some exceeded one million tokens.

The median conversation came in at around 72 thousand tokens.

The distribution is quite telling. In that median, about 2 thousand tokens came from user messages and agent responses.

Tool results, in turn, accounted for approximately 85% of the content. Moreover, file reads and shell commands dominated that volume.

The adopted solution applies a budget of 8 thousand tokens per trajectory. Priority starts with user messages and agent responses.

Then come intermediate model messages, tool results, and system messages.

However, messages above the limit are cut off. The ones that follow are left out.

The company acknowledged the limitation. However, detecting behavior such as ignoring a tool error requires access to the corresponding result, which usually sits at low priority.

According to the company, query-dependent context selection remains under investigation.

The foundation built on Trajectories

The feature builds on a function introduced in September. LangSmith Trajectories shows interactions, tool calls, and subagent activity in chronological order.

After locating a session, the developer reviews the trajectory. Then, they access the traces to inspect operations, attempts, and execution details.

The example used in the previous announcement illustrates this well. A support agent was reusing an outdated tool result, and the trajectory showed where the flow diverged from what was expected.

The warning about sensitive data in traces

This point deserves the attention of anyone instrumenting agents. Moreover, OpenTelemetry's generative AI conventions cover input messages, output messages, and tool call arguments.

This content can carry personal or confidential information. Consequently, logging everything requires control over telemetry collection and handling.

The company itself points to detecting exposed data as one application of the search. Still, finding information that has already been logged differs from preventing it from being logged in the first place.

Where to test it today

The feature is in public beta on LangSmith Cloud. Moreover, availability covers the United States, European Union, and APAC regions on Google Cloud.

Access comes at no additional cost. Organization admins enable the feature in the settings.

Follow our profile on Instagram!