Nielsen Norman Group study finds AI designs better with synthesized research than with raw data

- Reading
- 5 min
A Nielsen Norman Group study compared AI-generated designs based on raw user comments with designs generated from the same research after it was synthesized into findings and recommendations. Synthesized research won in both tests, with an advantage of up to 11.4 percentage points.
The Nielsen Norman Group published a study on November 20, 2026, conducted by Megan Chan and Rachel Banawa, that tackles a practical question for anyone already using generative AI in the design workflow: is it worth synthesizing research before feeding it to the model, or can you just throw in raw user comments and let the AI decide what matters? According to the experiment, the answer is that the synthesis step is still worth the effort.
The test: two scenarios, two sources of context
The researchers asked an AI model to generate interfaces for two design tasks: a single-page form to rebook a canceled flight and a multi-step checkout flow for an online store. The prompt and the design system information were always the same; what changed was the type of research context given to the model.
There were two conditions:
- Raw user comments: about 100 real comments per task, with no processing whatsoever.
- Synthesized findings and recommendations: the same comment base, but after a researcher analyzed it with an AI tool and decided which findings to include and what to recommend.
Because AI tools are non-deterministic, the team generated 8 designs per task and per condition. A human evaluator and an LLM judge scored each design, blind, against a predefined scorecard (for example, "every checkout step shows the items being purchased"). The two evaluators agreed on about 96% of the scoring decisions, which lends some confidence to the results.
The numbers: an 8.4 percentage point gap
In both tasks, designs generated from synthesized findings and recommendations met more scorecard criteria than those generated from raw comments: 77.9% versus 69.5%, on average, a statistically significant 8.4 percentage point difference (p < 0.001). The gain was not uniform across tasks.
- In the flight rebooking task, the synthesized research advantage was 11.4 percentage points.
- In the checkout task, it was 5.5 percentage points, smaller, but in the same direction.
The flight rebooking case shows what that difference means in practice. Some raw comments complained about losing loyalty miles during rebooking, but none of the designs generated from raw comments included information about the impact on miles; every design generated from synthesized research did. In the checkout task, several comments complained that items disappeared from the cart during the process: fewer than half of the raw-data-based designs kept the items and the total visible at every step, compared with nearly all of the ones based on synthesized research.
The smaller gain in checkout has an explanation: the task prompt itself already described a first-time buyer, with a discount coupon, needing the gift within two days, which already signaled part of the problems the comments also raised. When the prompt does part of the interpretation work, there's less room for synthesis to make a difference.
Why raw data doesn't get there alone
The researchers' explanation is straightforward: raw data describes problems, not solutions. A comment like "the items disappeared from my cart" names a frustration but doesn't say what to build in response. And even when a user suggests a fix, that suggestion isn't a reliable guide: people are good at describing problems and bad at predicting what they'd actually use.
Without the synthesis layer, it's the AI that has to decide on its own what matters and what change to make, a difficult call because, in practice, user feedback is just one input among many. Designers weigh feedback against business needs, technical constraints, product goals, and what the team has already tried. AI tools don't handle that kind of trade-off well without human guidance.
Generating an interface is just one part of design. Understanding which user problems matter for your product and translating that understanding into design recommendations remain an essential contribution of UX professionals, whether the designs are created by humans or AI.
Megan Chan and Rachel Banawa, researchers at the Nielsen Norman Group
What this changes for whoever decides what the team builds
For product designers and PMs who already put AI tools in the middle of the discovery workflow, the takeaway is concrete: the bottleneck isn't the model's capacity to process volumes of data, it's the quality of the interpretation that reaches it. The researchers call this UX-context design: the practice of selecting, organizing, and maintaining the UX knowledge that guides an AI-generated design, as a second audience alongside the people on the team.
In practice, this changes the order of steps in a project with AI in the middle. Before pasting an interview transcript or an app store comment export into a prompt, it's worth asking whether someone has already turned that into a finding (what the data means) and a recommendation (what to do about it). Skipping that step doesn't save work: it shifts the interpretation decision onto the AI, which has less business context to make that choice well.
This also repositions the role of the UX researcher on a team that relies heavily on AI. Research synthesis stops being just an artifact for the team to read; it becomes a direct input into the quality of what the AI delivers, measurable in scorecard accuracy percentage points.
What's still left to answer
The authors themselves place limits on the result. The dataset was small, about 100 short comments per task, and probably cleaner than real-world feedback: more than half of the comments in each task supported some finding, while research on app store reviews (cited in the study) found usefulness in only about a third of reviews. The two chosen tasks, a form and a checkout, are also design patterns a model has likely already seen many times.
The researchers expect the advantage of synthesized research to grow with larger, messier datasets and with design problems less familiar to the model.
Also left open: whether the gain is limited to the specific points covered by the findings or spills over into other aspects of the design; whether the result holds with interview transcripts, usability test notes, or survey responses; whether other AI models reproduce the pattern; and how much of the benefit comes from synthesis itself versus explicit design recommendations. NN/g has already signaled it plans to investigate these questions in future studies, including separating the effect of findings from that of recommendations.
Translated from the Brazilian Portuguese original · Read the original
Which Qualitative Research Method to Use at Each Product Stage
A practical guide based on material from the Nielsen Norman Group for deciding between usability testing, interviews, field studies, and diary studies, with what changes when applying this with a real team in Brazil.





