Search Quality: Evaluating and Tuning Relevance
Search quality is the ongoing discipline of measuring whether search results are actually good — not just whether the system runs — and using that measurement to guide relevance tuning over time.
Quality is not the same as uptime
A search system can be fast and error-free while still returning poor matches. Search quality specifically addresses whether the results are useful, which requires a different kind of measurement than standard system monitoring.
Two ways to measure quality
Human relevance judgments — rating a sample of query-result pairs against a rubric — and behavioral signals, like click-through rate, add-to-cart rate, and reformulation rate. Each has blind spots alone, which is why most teams use both together.
Query reformulation as a quality signal
When a shopper immediately searches again with a different, related query, that's a strong signal the first attempt didn't return what they wanted — even if it technically returned some results.
A/B testing search changes safely
Because relevance changes affect every query at once, testing changes on a subset of traffic before a full rollout is the standard way to catch regressions before they affect the whole store.
Common questions
How is search quality different from search analytics?
Analytics is the raw metrics. Quality is the evaluative discipline of judging whether those metrics reflect good outcomes, often combining metrics with direct human review.
What is query reformulation, and why does it matter?
It's when a shopper searches again shortly after an initial query, usually with different wording — a strong sign the first search didn't satisfy them, even without a formal survey.
Why test search changes on a subset of traffic first?
Because a relevance change affects every query running through it at once, so a subset test limits the blast radius if the change turns out to be a regression rather than an improvement.