arXiv preprint ·
LLM-based User-Product Relevance Evaluation for Dynamic Product Advertising
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Dynamic Product Advertising (DPA) retrieval systems are a core component of modern e-commerce advertising; however, their evaluation remains challenging. Conventional offline engagement metrics are limited by sparse, biased feedback, while online evaluation is costly and time-consuming. In this work, we propose a user--product relevance metric for offline evaluation of DPA retrieval systems. The metric first compresses the engagement history into a compact set of intent representatives via Semantic-ID-grouped, event-stratified segmentation, then assesses the relevance of a recommended product to each individual intent and aggregates them as a user-level relevance score, yielding a stable and interpretable measure of retrieval quality. To compute this metric at scale, we leverage large language models (LLMs) as semantic relevance judges, capturing nuanced functional and semantic relationships that heuristic methods often miss. We propose an agentic, human-aligned prompting framework that generates multi-category relevance guidelines and distills teacher-model judgments into an efficient student model for scalable evaluation. The resulting metric strongly correlates with offline and online business metrics, enabling faster and more reliable DPA retrieval iteration before A/B testing.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Information Retrieval
Co-adaptive Rule Routing and Constraining for Structured Knowledge Retrieval
Retrieving Complementary Evidence for Multi-Hop RAG with Semantic Gain Graphs
RES-MR: Risk-Aware Reasoning for Explainable and Safe Medication Recommendation
Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations
EVISTA: Unified Multimodal Video Search with Evidence-Aware Temporal Path Optimization
Get the few papers worth your time in Information Retrieval every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.