arXiv preprint ·
Measuring User Success in Randomized Search Experiments: Interpreting Good Abandonment in Generative AI Search
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Randomized search experiments increasingly evaluate experiences in which large language models answer information needs directly on the results page. Click status alone cannot identify success: a click may lead to an unsatisfactory result, while no click may indicate successful answer consumption, or "good abandonment." Prior research has shown that good abandonment is difficult to detect because dwell, scrolling, and reformulation are meaningful only in context. We present Session Success Rate (SSR), a human-grounded, interpretable outcome metric designed for online controlled experiments. Trained judges label complete session experiences using privacy-protected interaction evidence and video recordings; these offline labels train models scored online from query events, viewport behavior, result-page interactions, and semantic reformulations. Out-of-sample evaluation on 6,904 labeled queries yields 0.75 balanced accuracy overall and 0.68 within abandoned queries; session-level balanced accuracy is 0.68. The sample shows why interaction status is not an adequate outcome: 53.2% of abandonments were successful and 6.5% of clicked queries were unsuccessful. In a validation portfolio of seven ranking and five non-ranking experiments, SSR agreed with Sessions per User in seven cases, was inconclusive in four, and disagreed in one under a common p < 0.05 rule. Separately, across approximately 4,000 production experiments, SSR produced about 2.5 times the binary sensitivity of Sessions per User; in the studied workflow, this supported lower traffic allocations and greater experiment throughput under a fixed traffic budget.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Information Retrieval
"PRISM" Is Not All You Need: Cross-Topic Name Collisions and Scientific Paper Retrieval
Multi-Teacher Reasoning Distillation for LLM Recommendation with Self-Exploration
CCD: Context-Calibrated Open-Entity Out-of-Distribution Detection in In-Context Tabular Foundation Models
RecRSI: Mechanism-Guided Discovery of Recommendation Algorithms
Federated Vector Retrieval across Heterogeneous VectorDBs
Get the few papers worth your time in Information Retrieval every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.