arXiv preprint ·
Robustness Analysis of Pareto-based Joint Evaluation of Fairness and Relevance in Recommender Systems
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Recommender Systems (RSs) can be evaluated for many aspects, two of them being recommendation relevance and fairness. These two aspects can be evaluated either in isolation, using a measure for each aspect; or jointly with a single`relevance-aware' fairness measure that quantifies fairness with respect to relevance. Yet, neither approach provides a simultaneous evaluation of both relevance and fairness: the first approach may result in two different rankings of models, each corresponding to an evaluation aspect; the second approach often provides an imbalanced evaluation of the two aspects. To address this gap, we propose Distance to Pareto Frontier (DPFR), a new approach to jointly evaluate RS fairness and relevance. DPFR is based on a Pareto frontier of relevance and fairness scores from existing measures. The frontier is generated from user-item interaction data and represents various Pareto-optimal trade-offs between the two aspects. Thus, the distance from a point in the frontier provides a joint score of fairness and relevance. As DPFR can be used with existing measures, it is modular and intuitive. Our experiments with 4 recommenders and 3 fairness-aware re-rankers on 6 real-world datasets reveal inconsistent associations between existing measures and DPFR, showing that they are not reliable proxies for evaluations based on Pareto optimality. We further examine the robustness of DPFR as a joint evaluation approach for fairness and relevance by conducting experiments with artificial recommendations and with various fairness-relevance trade-offs, as well as by analysing DPFR's agreement to simple measures of fairness and relevance. These analyses show that DPFR is more robust than existing joint evaluation approaches. Finally, we provide guidelines for evaluating fairness and relevance jointly with DPFR. Our code: https://github.com/theresiavr/robustness-DPFR-recsys-evaluation.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Information Retrieval
Baseline and Incremental: User, Item, and Request Debiasing in Instagram Feed Ranking
Our Own Jev: Typed Decisions in One Forward Pass, from an In-House Judge to Open LoRA Models
Factual Access States in Diffusion Language Models: From Mechanism to Knowledge Editing and Retrieval
Progressive Learning from Content to Relevance to Personalization for CTR Prediction
GARRET: Generative Feedback for Region Retrieval in Visual Documents
Get the few papers worth your time in Information Retrieval every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.