Product search needs a relevance contract before a smarter model

A practical evaluation design for separating exact product answers from alternatives, accessories and plausible mistakes.

Product search needs a relevance contract before a smarter model

A shopper types “replacement lid for a one-litre steel flask”. A search engine returns a beautiful new flask, a lid for another diameter, and a cleaning brush. Every result is related to the words. None necessarily completes the job. For a retailer, the useful AI question is therefore specific: can the search system distinguish the requested item from an alternative, an accessory and an attractive mistake?

This analysis draws on product-search research published in 2022 and 2024, checked on 17 October 2026. Its evidence includes Amazon search data from the United States, Spain and Japan and Wayfair's home-products benchmark. Those settings do not establish performance for another catalogue, language or country. The worked example and evaluation process below are editorial proposals, not reported retailer results.

Three research lessons that should not be merged into one score

Amazon's 2022 ESCI benchmark separates exact matches, substitutes, complements and irrelevant results. It includes English, Spanish and Japanese query-product judgments. The paper also reports annotation ambiguity around whether a mismatched attribute makes an item a substitute or irrelevant. Its ranking evaluation assigns different gains to the four classes. These are useful research choices, rather than a universal definition of customer value. [Shopping Queries Dataset paper](https://arxiv.org/html/2206.06588v1)

Wayfair's WANDS release uses a different scheme: exact, partial and irrelevant. It provides query, product and judgment files, including product descriptions and structured features. Its 2022 home-products setting offers a second way to examine relevance, but a “partial” label cannot simply be treated as an ESCI substitute without reviewing the annotation meaning. [Wayfair WANDS repository](https://github.com/wayfair/WANDS)

COSMO, described by Amazon in 2024, addresses another layer: knowledge about shopping intentions. Its pipeline generates candidate commonsense assertions with language models and refines them using classifiers trained with human annotations. The authors explicitly acknowledge noise in generated knowledge. Their account of Amazon's own system does not establish that inferred intentions are verified facts about every product. [COSMO author publication record](https://www.amazon.science/publications/cosmo-a-large-scale-e-commerce-common-sense-knowledge-generation-and-serving-system-at-amazon)

The synthesis is practical. A model can help connect “something for a small hallway” to a shopping purpose. A separate relevance policy must still decide whether a particular product meets the request. Structured evidence must establish dimensions, compatibility and availability. Increasing the model's fluency does not settle those other decisions.

Write down what a result is allowed to promise

For the fictional flask search, a merchandising specialist and a search engineer could agree on this contract before comparing models:

| Candidate | Required evidence | Presentation decision | | --- | --- | --- | | Replacement lid for the specified flask | Matching model or documented dimensions and attachment | Eligible for the main results | | Another flask with its own lid | It solves a different purchasing task | Show separately only if the shopper wants alternatives | | Cleaning brush | Useful alongside the flask | Accessory area, not an exact answer | | Lid with missing compatibility data | Compatibility unresolved | Do not describe it as compatible |

These categories are deliberately local to the shop's task. They are not a claim that a research dataset uses this exact example. The important decision is what the business considers disqualifying. Missing compatibility cannot be repaired by a high semantic similarity score.

Keep the product record and the interpretation separate. The record might state diameter, thread type, material and model identifiers, each with provenance. A query interpreter might propose a candidate category or ask which flask the shopper owns. If that proposal cannot be resolved, a short clarifying question is more useful than a confident compatibility statement.

Build a test that exposes the expensive mistakes

Start with a bounded sample of actual shopping tasks, handled under the retailer's privacy rules. Remove unnecessary personal information. Include exact identifiers, broad exploratory searches, misspellings, local terminology, out-of-stock items and products with incomplete attributes. Avoid evaluating only the popular queries that are already easy.

Have two reviewers label a subset independently. Record disagreements and the reason for each decision. “Not my preferred style” is different from “does not fit the stated connector”. Resolve the policy before training on the labels; otherwise the system will learn inconsistent instructions and the headline score will conceal them.

Keep a holdout set of queries and recently introduced products away from tuning. Compare the proposed search system with the existing search on the same catalogue snapshot. Save the returned product identifiers and order, so that a later price or description change does not rewrite the experiment. This also makes it possible to distinguish a retrieval failure from bad catalogue data.

Measure more than an overall ranking score. For this example, report how often the first screen contains a verified exact match, how often incompatible lids appear as exact answers, how often all exact matches are missed, and how often the system correctly asks for clarification. Report results separately for known model identifiers and vague requests. An average improvement can coexist with a worse compatibility failure rate.

Follow a result through to the shopping task

Suppose the revised system retrieves the correct lid more often but customers still abandon the page. Review the journey before blaming the ranker. Is the compatibility evidence visible? Is the correct variant selectable? Is the item deliverable to the shopper's location? Does the page quietly change the selected diameter? These are different failure points and need different fixes.

A limited experiment could compare task completion and wrong-item feedback while preserving a way back to the existing search. Record which system and catalogue version served each eligible test request without collecting unnecessary identity data. Define a stop condition for a rise in compatibility complaints. The article proposes this experiment; it does not predict a conversion or revenue increase.

The potential benefit is more defensible search: fewer plausible distractions and clearer handling of missing information. The costs include maintaining product attributes, adjudicating labels, serving additional model steps and reviewing failures. A retailer with poor compatibility data may gain more from fixing those records than from replacing its model.

Begin with one product family where a wrong answer can be identified reliably. Establish the relevance contract, repair the attributes it requires, test against a frozen baseline and inspect the harmful errors separately. Expand only when those results support the next product family and language. Search quality is earned at the boundary between what the shopper asks and what the catalogue can prove.