Robustness and User-Perceived Value of Popularity Calibration in Music Recommendation: A User Study

• URL: http://arxiv.org/abs/2608.05402v1
• Date: Wed, 05 Aug 2026 20:44:34 GMT
• Title: Robustness and User-Perceived Value of Popularity Calibration in Music Recommendation: A User Study
• Abstract summary: We study the perceived value and measurement reliability of popularity calibration in music recommendation.<n>Our results show that users perceive differences in popularity composition, but do not clearly prefer lists.<n>These findings contribute to a more critical understanding of popularity calibration as both an offline metric and a user-facing construct.
• Score: 10.734672773406439
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Popularity calibration in recommender systems has been studied both as a form of user-centered personalization and as an indicator of popularity bias. Most existing work evaluates calibration through offline metrics, often assuming that users prefer recommendation lists whose popularity distribution matches their historical consumption profile. However, user studies on calibration remain limited, and existing findings suggest that calibrated recommendations do not necessarily have a strong effect on user experience. Moreover, although prior work has shown that calibration metrics can correlate with users' perceptions of recommendation lists, the robustness of this relation remains unclear under different levels of item familiarity and incomplete user-history information. In this work, we study the perceived value and measurement reliability of popularity calibration in music recommendation. We construct personalized track lists from users' recent listening histories and use a controlled naive recommender to create lists with different popularity compositions: highpop-heavy, lowpop-heavy, and calibrated. We investigate whether users perceive differences between these lists, whether calibrated lists are preferred, how robust JSD-based popularity calibration is under different familiarity and history-availability conditions, and how computational popularity labels align with users' own popularity judgments. Our results show that users perceive differences in popularity composition, but do not clearly prefer calibrated lists. We further find that the relation between JSD and perceived popularity depends on item familiarity, list composition, and available user history, while computational and user-judged popularity labels only weakly align. These findings contribute to a more critical understanding of popularity calibration as both an offline metric and a user-facing construct.

Related papers

• Aligning Recommendations with User Popularity Preferences [2.1976433151201658]
  Popularity bias is a pervasive problem in recommender systems, where recommendations disproportionately favor popular items.<n>This work studies popularity bias through the lens of user-recommender alignment.<n>We propose SPREE, an inference-time mitigation method for sequential recommenders based on activation steering.
  arXiv  Detail & Related papers  (2026-04-01T15:45:24Z)
• A Causal Information-Flow Framework for Unbiased Learning-to-Rank [52.54102347581931]
  In web search and recommendation systems, user clicks are widely used to train ranking models.<n>We introduce a novel causal learning-based ranking framework that extends Unbiased Learning-to-Rank.<n>Our method consistently reduces measured bias leakage and improves ranking performance.
  arXiv  Detail & Related papers  (2026-01-09T07:19:35Z)
• Scalable Utility-Aware Multiclass Calibration [53.28176049547449]
  Utility calibration is a general framework that measures the calibration error relative to a specific utility function.<n>We demonstrate how this framework can unify and re-interpret several existing calibration metrics.
  arXiv  Detail & Related papers  (2025-10-29T12:32:14Z)
• Addressing Personalized Bias for Unbiased Learning to Rank [56.663619153713434]
  Unbiased learning to rank (ULTR) aims to learn unbiased ranking models from biased user behavior logs.<n>We propose a novel user-aware inverse-propensity-score estimator for learning-to-rank objectives.
  arXiv  Detail & Related papers  (2025-08-28T14:01:31Z)
• Exploring the Effect of Context-Awareness and Popularity Calibration on Popularity Bias in POI Recommendations [1.389360509566256]
  Point-of-interest (POI) recommender systems help users discover relevant locations, but their effectiveness is often compromised by popularity bias.<n>This paper addresses this challenge by evaluating the effectiveness of context-aware models and calibrated popularity techniques as strategies for mitigating popularity bias.
  arXiv  Detail & Related papers  (2025-07-04T11:56:11Z)
• Taming Recommendation Bias with Causal Intervention on Evolving Personal Popularity [21.645389873171908]
  Popularity bias occurs when popular items are recommended far more frequently than they should be.<n>Existing debiasing methods mitigate popularity bias often uniformly across all users.<n>We propose a novel method called CausalEPP (Causal Intervention on Evolving Personal Popularity) for taming recommendation bias.
  arXiv  Detail & Related papers  (2025-05-20T12:59:16Z)
• Enhancing Sequential Music Recommendation with Personalized Popularity Awareness [56.972624411205224]
  This paper introduces a novel approach that incorporates personalized popularity information into sequential recommendation. Experimental results demonstrate that a Personalized Most Popular recommender outperforms existing state-of-the-art models.
  arXiv  Detail & Related papers  (2024-09-06T15:05:12Z)
• Going Beyond Popularity and Positivity Bias: Correcting for Multifactorial Bias in Recommender Systems [74.47680026838128]
  Two typical forms of bias in user interaction data with recommender systems (RSs) are popularity bias and positivity bias. We consider multifactorial selection bias affected by both item and rating value factors. We propose smoothing and alternating gradient descent techniques to reduce variance and improve the robustness of its optimization.
  arXiv  Detail & Related papers  (2024-04-29T12:18:21Z)
• Improving Recommendation System Serendipity Through Lexicase Selection [53.57498970940369]
  We propose a new serendipity metric to measure the presence of echo chambers and homophily in recommendation systems. We then attempt to improve the diversity-preservation qualities of well known recommendation techniques by adopting a parent selection algorithm known as lexicase selection. Our results show that lexicase selection, or a mixture of lexicase selection and ranking, outperforms its purely ranked counterparts in terms of personalization, coverage and our specifically designed serendipity benchmark.
  arXiv  Detail & Related papers  (2023-05-18T15:37:38Z)
• A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations [6.694971161661218]
  We study how different genres affect the inconsistency of recommendation performance. We find that users with little interest in popular content receive the worst recommendation accuracy. Our experiments show that particular genres contribute to a different extent to the inconsistency of recommendation performance.
  arXiv  Detail & Related papers  (2023-03-01T10:39:58Z)
• User-centered Evaluation of Popularity Bias in Recommender Systems [4.30484058393522]
  Recommendation and ranking systems suffer from popularity bias; the tendency of the algorithm to favor a few popular items while under-representing the majority of other items. In this paper, we show the limitations of the existing metrics to evaluate popularity bias mitigation when we want to assess these algorithms from the users' perspective. We present an effective approach that mitigates popularity bias from the user-centered point of view.
  arXiv  Detail & Related papers  (2021-03-10T22:12:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.