Rank-Preference Consistency as the Appropriate Metric for Recommender Systems

• URL: http://arxiv.org/abs/2404.17097v1
• Date: Fri, 26 Apr 2024 01:11:07 GMT
• Title: Rank-Preference Consistency as the Appropriate Metric for Recommender Systems
• Authors: Tung Nguyen, Jeffrey Uhlmann,
• Abstract summary: We argue that unitary-invariant measures of recommender system (RS) performance fail to assess fundamental RS properties. We propose rank-preference consistency, which simply counts the number of prediction pairs that are inconsistent with the user's expressed product preferences.
• Score: 4.3166389349316425
• License:
• Abstract: In this paper we argue that conventional unitary-invariant measures of recommender system (RS) performance based on measuring differences between predicted ratings and actual user ratings fail to assess fundamental RS properties. More specifically, posing the optimization problem as one of predicting exact user ratings provides only an indirect suboptimal approximation for what RS applications typically need, which is an ability to accurately predict user preferences. We argue that scalar measures such as RMSE and MAE with respect to differences between actual and predicted ratings are only proxies for measuring RS ability to accurately estimate user preferences. We propose what we consider to be a measure that is more fundamentally appropriate for assessing RS performance, rank-preference consistency, which simply counts the number of prediction pairs that are inconsistent with the user's expressed product preferences. For example, if an RS predicts the user will prefer product A over product B, but the user's withheld ratings indicate s/he prefers product B over A, then rank-preference consistency has been violated. Our test results conclusively demonstrate that methods tailored to optimize arbitrary measures such as RMSE are not generally effective at accurately predicting user preferences. Thus, we conclude that conventional methods used for assessing RS performance are arbitrary and misleading.

Related papers

• Preference Diffusion for Recommendation [50.8692409346126]
  We propose PreferDiff, a tailored optimization objective for DM-based recommenders. PreferDiff transforms BPR into a log-likelihood ranking objective to better capture user preferences. It is the first personalized ranking loss designed specifically for DM-based recommenders.
  arXiv  Detail & Related papers  (2024-10-17T01:02:04Z)
• Quantifying User Coherence: A Unified Framework for Cross-Domain Recommendation Analysis [69.37718774071793]
  This paper introduces novel information-theoretic measures for understanding recommender systems. We evaluate 7 recommendation algorithms across 9 datasets, revealing the relationships between our measures and standard performance metrics.
  arXiv  Detail & Related papers  (2024-10-03T13:02:07Z)
• Belief-State Query Policies for Planning With Preferences Under Partial Observability [18.821166966365315]
  Planning in real-world settings often entails addressing partial observability while aligning with users' preferences. We present a novel framework for expressing users' preferences about agent behavior in a partially observable setting using parameterized belief-state query (BSQ) preferences. We show that BSQ preferences provide a computationally feasible approach for planning with preferences in partially observable settings.
  arXiv  Detail & Related papers  (2024-05-24T20:04:51Z)
• Beyond Static Calibration: The Impact of User Preference Dynamics on Calibrated Recommendation [3.324986723090369]
  Calibration in recommender systems is an important performance criterion. Standard methods for mitigating miscalibration typically assume that user preference profiles are static. We conjecture that this approach can lead to recommendations that, while appearing calibrated, in fact, distort users' true preferences.
  arXiv  Detail & Related papers  (2024-05-16T16:33:34Z)
• Going Beyond Popularity and Positivity Bias: Correcting for Multifactorial Bias in Recommender Systems [74.47680026838128]
  Two typical forms of bias in user interaction data with recommender systems (RSs) are popularity bias and positivity bias. We consider multifactorial selection bias affected by both item and rating value factors. We propose smoothing and alternating gradient descent techniques to reduce variance and improve the robustness of its optimization.
  arXiv  Detail & Related papers  (2024-04-29T12:18:21Z)
• Off-Policy Evaluation of Ranking Policies under Diverse User Behavior [25.226825574282937]
  Inverse Propensity Scoring (IPS) becomes extremely inaccurate in the ranking setup due to its high variance under large action spaces. This work explores a far more general formulation where user behavior is diverse and can vary depending on the user context. We show that the resulting estimator, which we call Adaptive IPS (AIPS), can be unbiased under any complex user behavior.
  arXiv  Detail & Related papers  (2023-06-26T22:31:15Z)
• Item-based Variational Auto-encoder for Fair Music Recommendation [1.8782288713227568]
  The EvalRS DataChallenge aims to build a more realistic recommender system considering accuracy, fairness, and diversity in evaluation. Our proposed system is based on an ensemble between an item-based variational auto-encoder (VAE) and a Bayesian personalized ranking matrix factorization (BPRMF)
  arXiv  Detail & Related papers  (2022-10-24T06:42:16Z)
• Recommendation Systems with Distribution-Free Reliability Guarantees [83.80644194980042]
  We show how to return a set of items rigorously guaranteed to contain mostly good items. Our procedure endows any ranking model with rigorous finite-sample control of the false discovery rate. We evaluate our methods on the Yahoo! Learning to Rank and MSMarco datasets.
  arXiv  Detail & Related papers  (2022-07-04T17:49:25Z)
• Estimating and Penalizing Induced Preference Shifts in Recommender Systems [10.052697877248601]
  We argue that system designers should: estimate the shifts a recommender would induce; evaluate whether such shifts would be undesirable; and even actively optimize to avoid problematic shifts. We do this by using historical user interaction data to train predictive user model which implicitly contains their preference dynamics. In simulated experiments, we show that our learned preference dynamics model is effective in estimating user preferences and how they would respond to new recommenders.
  arXiv  Detail & Related papers  (2022-04-25T21:04:46Z)
• Control Variates for Slate Off-Policy Evaluation [112.35528337130118]
  We study the problem of off-policy evaluation from batched contextual bandit data with multidimensional actions. We obtain new estimators with risk improvement guarantees over both the PI and self-normalized PI estimators.
  arXiv  Detail & Related papers  (2021-06-15T06:59:53Z)
• Set2setRank: Collaborative Set to Set Ranking for Implicit Feedback based Recommendation [59.183016033308014]
  In this paper, we explore the unique characteristics of the implicit feedback and propose Set2setRank framework for recommendation. Our proposed framework is model-agnostic and can be easily applied to most recommendation prediction approaches.
  arXiv  Detail & Related papers  (2021-05-16T08:06:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.