Before you measure your AI product, you need to find where it fails

Every team building a product with generative AI eventually reaches the question: how do we know if this is working? The industry's standard answer has been "evals": automated tests that check whether your AI system's responses remain good after a change to a prompt, model, or code. Mike Krieger, Anthropic's former CPO and now head of the company's Labs, went so far as to say that "if there's one thing you can teach product people, it's that writing evals is probably the most important thing right now". Garry Tan, CEO of Y Combinator, made a similar point: evals are becoming the real competitive moat for AI startups.

The problem, according to Hamel Husain and Shreya Shankar in a piece published in Lenny Rachitsky's newsletter, isn't the idea of measuring. It's the order in which teams do it. After working with more than 50 AI companies, the two noticed a pattern: almost everyone jumps straight to writing metrics and skips the previous step, which they call error discovery. The name is new, but the concept is an evolution of what Husain and Shankar's own earlier piece called "error analysis". The name change isn't cosmetic: the stated goal now is to identify which failures are worth measuring, not to catalog every possible failure.

Why this is product discovery, not engineering

The analogy the authors use is direct: just as product discovery (that research phase that shows which problems are worth solving, before any line of the roadmap) reveals what to build, error discovery reveals what to measure. Without this step, the team builds dashboards around generic metrics that push the product in the wrong direction.

The risk of skipping this phase is writing metrics too early, based on assumptions about what matters. And here comes a phenomenon the authors call criteria drift: your criteria for what counts as "a good response" shift as you review real examples. You only discover certain failure patterns after looking at the data, not before. This explains why throwing a bunch of traces (the complete records of a user session with the AI product) at a coding agent and asking it to "find the problems" isn't enough: the agent is quick to spot obvious errors, but it doesn't know, beforehand, what your definition of a good product is.

The case that illustrates everything: a "success" that was actually a product failure

The piece's most illustrative example comes from work with Nurture Boss, an AI leasing assistant that helps property managers talk to prospective tenants. In a real interaction, the applicant writes: "That's outside my budget. Thanks for your time." The assistant replies: "You're welcome! If your situation changes or you have other questions, just reach out. Have a great day!"

To most AI agents evaluating this conversation, it looks like a success: the response is polite, it resolved the interaction, no one complained. But the product's goal is to close deals, which includes offering alternatives when the applicant runs straight into a budget constraint. The assistant should have suggested cheaper units or other properties from the same management company. Only after manually reviewing this trace did Husain and Shankar arrive at "objection handling" as a criterion worth measuring. Had they asked the agent to check for objection handling from the start, it would have caught the error on its own. The problem is that no one knew, a priori, that this was a criterion to check for.

In a broader study with 100 production traces from the same assistant, the authors compared automated eval tools and coding agents against human review. The agents got it wrong precisely in cases that require product judgment and context beyond the trace, such as broken Markdown formatting in text messages and missed handoffs to humans (in addition to objection handling itself). On the other hand, the agents were good at catching obvious failures within the trace, such as responses that contradict a tool's output. And, curiously, they also found problems that humans missed, at the cost of generating noise by flagging good responses as failures.

The three-step process (with a coding agent)

The proposed solution combines automation with human review using an active learning principle: choosing the most informative examples to review, given limited time. In practice, the workflow described in the piece has three steps:

1. Gather traces. Each trace needs to contain the user input, system prompt, tool calls, and the final output, enough for someone to reconstruct what happened. If the product doesn't have instrumentation yet, you can ask a coding agent to log this in JSONL. Without real users yet, you can simulate synthetic queries by defining dimensions (task, user type, request clarity) and generating one model call per combination, avoiding having the agent produce homogeneous variations in a single shot.

1. Review and annotate. The authors published an open-source plugin for this: npx skills add https://github.com/ai-evals-course/evals-skills. Pointing the agent to the evals-start skill, it reads a sample of your data, learns the schema, and builds a customized local review interface, different for a leasing chat versus a writing assistant, for example. The rule is to annotate at least 10 traces in free text before letting the agent suggest problems, as a safeguard against automation bias. Good annotation is specific ("the assistant said the unit was available when the tool showed it was rented"), not a generic verdict ("bad response"). The goal is to reach about 100 annotated traces before considering the step mature.

1. Turn patterns into product priorities. With the annotated base, the agent groups the annotations into failure modes and counts the frequency of each one, turning it into a prioritized list of what to tackle first on the roadmap.

Who has already reaped results from this

The piece cites concrete cases of companies that invested in this discipline before writing metrics. Shopify used evals to guide the development of an AI workflow builder that ended up 2.2 times faster and 68% cheaper than the frontier-model system it replaced. Cursor developed the routing for its Auto Balance using evals, resulting in much higher user satisfaction and a 41% cost reduction. Ramp raised the accuracy of its automatic receipt-collection product from 35% to 83% after investing in evals. Harvey rebuilt its AI contract reviewer using the same process, nearly doubling its internal product quality score.

What remains open

The piece itself acknowledges the limit: synthetic data doesn't replace real users, and the learning curve of the human-agent process requires several iterations before the agent's suggestions become reliable. It's also worth noting that this is paid newsletter content, so the final step (turning failure clusters into actual metrics) is only sketched out in the open excerpt. Even so, the takeaway that already stands out is clear: before writing any evaluation metric for an AI feature in your product, it's worth asking whether someone on the team has already sat down to read, one by one, a diverse sample of real traces and annotate, in plain language, what bothers them about each response. If the answer is no, the metric that comes out of it will probably measure the wrong thing.