Mitigating the Participation Bias by Balancing Extreme Ratings

• URL: http://arxiv.org/abs/2502.03737v1
• Date: Thu, 06 Feb 2025 02:58:46 GMT
• Title: Mitigating the Participation Bias by Balancing Extreme Ratings
• Authors: Yongkang Guo, Yuqing Kong, Jialiang Liu,
• Abstract summary: We consider a robust rating aggregation task under the participation bias. Our goal is to minimize the expected squared loss between the aggregated ratings and the average of all underlying ratings.
• Score: 3.5785450878667597
• License:
• Abstract: Rating aggregation plays a crucial role in various fields, such as product recommendations, hotel rankings, and teaching evaluations. However, traditional averaging methods can be affected by participation bias, where some raters do not participate in the rating process, leading to potential distortions. In this paper, we consider a robust rating aggregation task under the participation bias. We assume that raters may not reveal their ratings with a certain probability depending on their individual ratings, resulting in partially observed samples. Our goal is to minimize the expected squared loss between the aggregated ratings and the average of all underlying ratings (possibly unobserved) in the worst-case scenario. We focus on two settings based on whether the sample size (i.e. the number of raters) is known. In the first setting, where the sample size is known, we propose an aggregator, named as the Balanced Extremes Aggregator. It estimates unrevealed ratings with a balanced combination of extreme ratings. When the sample size is unknown, we derive another aggregator, the Polarizing-Averaging Aggregator, which becomes optimal as the sample size grows to infinity. Numerical results demonstrate the superiority of our proposed aggregators in mitigating participation bias, compared to simple averaging and the spectral method. Furthermore, we validate the effectiveness of our aggregators on a real-world dataset.

Related papers

• Can Large Language Models Understand Preferences in Personalized Recommendation? [32.2250928311146]
  We introduce PerRecBench, disassociating evaluation from user rating bias and item quality. We find that the LLM-based recommendation techniques that are generally good at rating prediction fail to identify users' favored and disfavored items when the user rating bias and item quality are eliminated. Our findings reveal the superiority of pairwise and listwise ranking approaches over pointwise ranking, PerRecBench's low correlation with traditional regression metrics, the importance of user profiles, and the role of pretraining data distributions.
  arXiv  Detail & Related papers  (2025-01-23T05:24:18Z)
• Analytical and Empirical Study of Herding Effects in Recommendation Systems [72.6693986712978]
  We study how to manage product ratings via rating aggregation rules and shortlisted representative reviews. We show that proper recency aware rating aggregation rules can improve the speed of convergence in Amazon and TripAdvisor.
  arXiv  Detail & Related papers  (2024-08-20T14:29:23Z)
• Going Beyond Popularity and Positivity Bias: Correcting for Multifactorial Bias in Recommender Systems [74.47680026838128]
  Two typical forms of bias in user interaction data with recommender systems (RSs) are popularity bias and positivity bias. We consider multifactorial selection bias affected by both item and rating value factors. We propose smoothing and alternating gradient descent techniques to reduce variance and improve the robustness of its optimization.
  arXiv  Detail & Related papers  (2024-04-29T12:18:21Z)
• Tensor-based Collaborative Filtering With Smooth Ratings Scale [0.0]
  We introduce the ratings' similarity matrix which represents the dependency between different values of ratings on the population level. It is possible to improve the quality of proposed recommendations by off-setting the effect of either shifted down or shifted up users' rates.
  arXiv  Detail & Related papers  (2022-05-10T17:55:25Z)
• Holistic Approach to Measure Sample-level Adversarial Vulnerability and its Utility in Building Trustworthy Systems [17.707594255626216]
  Adversarial attack perturbs an image with an imperceptible noise, leading to incorrect model prediction. We propose a holistic approach for quantifying adversarial vulnerability of a sample by combining different perspectives. We demonstrate that by reliably estimating adversarial vulnerability at the sample level, it is possible to develop a trustworthy system.
  arXiv  Detail & Related papers  (2022-05-05T12:36:17Z)
• Abnormal-aware Multi-person Evaluation System with Improved Fuzzy Weighting [0.0]
  We choose the two-stage screening method, which consists of rough screening and score-weighted Kendall-$tau$ Distance. We use Fuzzy Synthetic Evaluation Method(FSE) to determine the significance of scores given by reviewers as well as their reliability.
  arXiv  Detail & Related papers  (2022-05-01T03:42:43Z)
• Cross Pairwise Ranking for Unbiased Item Recommendation [57.71258289870123]
  We develop a new learning paradigm named Cross Pairwise Ranking (CPR) CPR achieves unbiased recommendation without knowing the exposure mechanism. We prove in theory that this way offsets the influence of user/item propensity on the learning.
  arXiv  Detail & Related papers  (2022-04-26T09:20:27Z)
• Integrating Rankings into Quantized Scores in Peer Review [61.27794774537103]
  In peer review, reviewers are usually asked to provide scores for the papers. To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking of the papers they have reviewed. There are no standard procedure for using this ranking information and Area Chairs may use it in different ways. We take a principled approach to integrate the ranking information into the scores.
  arXiv  Detail & Related papers  (2022-04-05T19:39:13Z)
• Universal Off-Policy Evaluation [64.02853483874334]
  We take the first steps towards a universal off-policy estimator (UnO) We use UnO for estimating and simultaneously bounding the mean, variance, quantiles/median, inter-quantile range, CVaR, and the entire cumulative distribution of returns.
  arXiv  Detail & Related papers  (2021-04-26T18:54:31Z)
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap [140.98628848491146]
  We introduce deconfounding scores, which induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. In particular, we show that this technique could be an attractive alternative to standard regularizations.
  arXiv  Detail & Related papers  (2021-04-12T18:50:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.