Study shows verification layers in LLMs only filter answers, not fix errors

A study published on November 10, 2026 by InfoQ, authored by Pushpdeep Mishra and Wejdan Bagais within the InfoQ Certified Architect Program, overturns a common assumption among those putting LLMs into production: that stacking verification layers on top of a model's output increases accuracy cumulatively. The authors tested six reliability layers, from the deterministic checker to the human reviewer, on top of the same natural-language extractor, and measured what each one actually changed in the results. The short answer: they sort the answers by confidence, but correct almost nothing.

The angle speaks directly to anyone building agents and pipelines with LLMs today, because the design tested (model reads, deterministic layer checks, second model judges, human decides what's left) is close to the standard architecture any team puts together when putting an LLM into production. The study shows where this pattern delivers real value and where it's reliability theater.

The theory that didn't hold up

The hypothesis tested was that the layers are additive: each one would catch the errors the previous one let through, so accuracy would rise with every layer added. That's the logic behind much of today's agent architectures, with self-correction, sampling-based voting (sampling and voting), a second model acting as judge, and refusal of an answer when confidence is low.

The layers sorted the answers by how far to trust them and corrected almost none.

Pushpdeep Mishra and Wejdan Bagais, InfoQ Certified Architect Program

To test this rigorously, the authors used CADEC (CSIRO Adverse Drug Event Corpus), a corpus that has been public since 2015 with 1,250 forum posts about two medications, with each adverse reaction marked as an exact text span and linked to a SNOMED CT code. Having a ground truth per mention makes it possible to compare each layer's verdict against the correct answer, instead of trusting what the layer says about itself.

What the tested pipeline looked like

The design avoids the most common mistake: the model is never asked for the code directly. It only cites the text span; a search engine with no language model at all (a deterministic lookup) scans 227,554 keyword-to-code lines, extracted from the July 31, 2026 Australian edition of SNOMED CT, using the granite-embedding:30m embedding run locally, and shows the model a menu with the 20 closest concepts for it to choose from.

The six layers tested on top of this extractor were:

• Vocabulary check (0 tokens): confirms whether the code exists and whether the span is actually in the post.
• Self-correction: feeds back to the model the errors the check proved wrong.
• Voting: runs the extractor three times and uses majority rule.
• Judge (second model): a smaller model evaluates whether the extractor's answer is correct.
• Refusal: holds back doubtful answers for human review.
• Human review: decides on what refusal held back.

What each layer delivered, and what it cost

The results per layer, measured across three rounds on 40 development documents, show a clear pattern: the two free layers delivered on what they promised; the three paid ones did not.

Voting even destroyed 2 to 5 correct answers per round, because the one voting is the very model that got it wrong: a vote carries no information the answer didn't already have. And the judge, which did discriminate better when it saw the same menu of options as the extractor (separation 3.4 to 4.2 times sharper than the free check), simply had its verdict ignored by the rest of the pipeline. Removing the three paid layers and keeping only refusal changed the outcome by a single record, in one of the three rounds.

Recognizing is easier than generating

The most direct finding for anyone designing prompts and pipelines: asking the model to recall an identifier off the top of its head is far worse than asking it to recognize the right option in a menu. The authors tested three ways of obtaining the SNOMED code: recalling from memory, naming the concept and searching, or choosing from a menu. Asking for the code from memory produced 13% to 18% of codes that don't exist in any version of SNOMED, and left up to 38% of records without an answer. Choosing from a menu performed ten times better.

In short: never ask the model for an identifier, ID or code from memory. Ask it to cite the span and choose among options that a deterministic lookup has already validated, and let the code come from the lookup, not from the model's generation.

Determinism: most repeat, a few don't

Another practical point for anyone who already assumed that temperature=0 guarantees reproducibility: four of the five model families tested (llama3.1:8b, mistral:7b-instruct, qwen3:8b, granite4:micro-h) reproduced their output byte for byte across three identical runs. The extractor chosen by the authors, gpt-oss:20b, was the only mixture-of-experts model among the five, and the only one that varied: it agreed on span and code in 70% of mentions across the three runs, even at zero temperature. It was kept anyway, for delivering 6.5 F1 points more than llama3.1:8b.

The practical lesson: measure your own pipeline's baseline variation, with at least three identical runs, before assuming any gain from a new layer. Half of what looks like improvement can be the model's own noise.

The ceiling no layer can break through

The most structural finding is that no layer above the extractor can recover a mention the extractor simply never read. Nearly half of the mentions annotated in CADEC were never cited the way the annotators marked them, because of sentence boundaries that stretch, isolated words that get ignored, and figures of speech read literally. Once the mention is found, the pipeline is reliable: the right concept reaches the menu in 93% of cases, and the model picks correctly in 81% of them. The bottleneck is in reading, not in vocabulary.

Domain-specific models didn't solve this: BioMistral-7B couldn't reliably return parseable answers, and using SapBERT as the search engine's encoder improved the menu but made the final answer worse. What worked was simpler: a worked example based on the corpus's own annotation conventions, and a rule for reactions negated in the text.

What changes for those putting LLMs into production

The study tested seven more corpora and five model families beyond CADEC, and four of the five lessons repeated across other data. For anyone architecting LLM systems today, the points that hold across different domains are:

1. Verification layers filter, they don't correct: measure yield (correct answers over the total), not just the accuracy of what passed, because accuracy alone always favors whoever discards more.
2. Deterministic checking (vocabulary, schema, compiler) proves an answer is wrong, never that it's right; test it against your own ground truth before trusting it, because on place-name corpora it was only right 7% to 26% of the time.
3. Sampling-based voting can destroy correct answers when the voter is the very model that got it wrong.
4. A separate judge only helps if someone or something reads its verdict; a judge whose output nobody consumes is cost without effect.
5. The ceiling of any pipeline is what the extractor finds on its first read; investing in a second read (another model reviewing what the first one missed) is the only path the authors cite for breaking through that ceiling, and they didn't test it.

For Brazilian teams running agents and information extractors in production, with Postgres, pgvector or any RAG stack, the practical takeaway is direct: spending tokens on self-correction and voting tends to be money thrown away if nobody measured the real gain against the model's own noise. The money goes further on a well-designed deterministic check and a clear rule for when to hold an answer back for human review, than on yet another layer of AI checking AI.