arXiv preprint ·
When the Label Ignores the Request: Auditing Policy-Selected Targets in Synthetic Conversational Music Recommendation
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.1145/3842413.3842427
- Original
- Read it at arXiv
- Topics
Abstract
Synthetic dialogues generated by LLM pipelines now serve as complete conversational-recommendation benchmarks: an LLM listener talks to an LLM recommender, and the track logged next in the conversation becomes the official label for each turn. These policy-selected labels make large-scale evaluation reproducible, but they are proxies for what the simulated user asked. We audit the one place where label and request are directly comparable: turns where the user asks for an exact song by name. In the RecSys Challenge 2026 TalkPlay benchmark, using visible dialogue and catalog metadata alone, we find that the official label contradicts the user's exact-song request in half of the audited development turns. This matters beyond one benchmark: naming the desired item is the dominant intent in real music search, where deployed systems avoid substituting an alternative for an exactly named item, on the premise that it costs satisfaction. A small training-time supplement closes most of the gap: adding catalog-resolved request-satisfying targets to a small fraction of training turns yields a 53.3% relative gain in nDCG@20 on the 43 conflict turns while leaving the official metric intact, verified against a matched control that detects the same requests but trains only on official labels.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Information Retrieval
Learning to Route in Visual Space via Multi-Step Embedding Retrieval
CANOPY: Adaptive-Granularity Evidence Compression for Multimodal RAG
PatchHolmes: Agentic Patch Retrieval via Listwise Selection
RPTune: Learned Context Curation for LLM Catalog Search
SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale
Get the few papers worth your time in Information Retrieval every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.