A Self-boosted Framework for Calibrated Ranking

• URL: http://arxiv.org/abs/2406.08010v1
• Date: Wed, 12 Jun 2024 09:00:49 GMT
• Title: A Self-boosted Framework for Calibrated Ranking
• Authors: Shunyu Zhang, Hu Liu, Wentian Bao, Enyun Yu, Yang Song,
• Abstract summary: Calibrated Ranking is a scale-calibrated ranking system that pursues accurate ranking quality and calibrated probabilistic predictions simultaneously. Previous methods need to aggregate the full candidate list within a single mini-batch to compute the ranking loss. We propose a Self-Boosted framework for Calibrated Ranking (SBCR)
• Score: 7.4291851609176645
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scale-calibrated ranking systems are ubiquitous in real-world applications nowadays, which pursue accurate ranking quality and calibrated probabilistic predictions simultaneously. For instance, in the advertising ranking system, the predicted click-through rate (CTR) is utilized for ranking and required to be calibrated for the downstream cost-per-click ads bidding. Recently, multi-objective based methods have been wildly adopted as a standard approach for Calibrated Ranking, which incorporates the combination of two loss functions: a pointwise loss that focuses on calibrated absolute values and a ranking loss that emphasizes relative orderings. However, when applied to industrial online applications, existing multi-objective CR approaches still suffer from two crucial limitations. First, previous methods need to aggregate the full candidate list within a single mini-batch to compute the ranking loss. Such aggregation strategy violates extensive data shuffling which has long been proven beneficial for preventing overfitting, and thus degrades the training effectiveness. Second, existing multi-objective methods apply the two inherently conflicting loss functions on a single probabilistic prediction, which results in a sub-optimal trade-off between calibration and ranking. To tackle the two limitations, we propose a Self-Boosted framework for Calibrated Ranking (SBCR).

Related papers

• Optimal Baseline Corrections for Off-Policy Contextual Bandits [61.740094604552475]
  We aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. We propose a single framework built on their equivalence in learning scenarios. Our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it.
  arXiv  Detail & Related papers  (2024-05-09T12:52:22Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systems [33.46891569350896]
  Cascade ranking is widely used for large-scale top-k selection problems in online advertising and recommendation systems. Previous works on learning-to-rank usually focus on letting the model learn the complete order or top-k order. We name this method as Adaptive Neural Ranking Framework (abbreviated as ARF)
  arXiv  Detail & Related papers  (2023-10-16T14:43:02Z)
• Optimizing Partial Area Under the Top-k Curve: Theory and Practice [151.5072746015253]
  We develop a novel metric named partial Area Under the top-k Curve (AUTKC) AUTKC has a better discrimination ability, and its Bayes optimal score function could give a correct top-K ranking with respect to the conditional probability. We present an empirical surrogate risk minimization framework to optimize the proposed metric.
  arXiv  Detail & Related papers  (2022-09-03T11:09:13Z)
• Joint Optimization of Ranking and Calibration with Contextualized Hybrid Model [24.66016187602343]
  We propose an approach that can Jointly optimize the Ranking and abilities (JRC) for short. JRC improves the ranking ability by contrasting the logit value for the sample with different labels and constrains the predicted probability to be a function of the logit subtraction. JRC has been deployed on the display advertising platform of Alibaba and has obtained significant performance improvements.
  arXiv  Detail & Related papers  (2022-08-12T08:32:13Z)
• Certified Error Control of Candidate Set Pruning for Two-Stage Relevance Ranking [57.42241521034744]
  We propose the concept of certified error control of candidate set pruning for relevance ranking. Our method successfully prunes the first-stage retrieved candidate sets to improve the second-stage reranking speed.
  arXiv  Detail & Related papers  (2022-05-19T16:00:13Z)
• Unbiased Pairwise Learning to Rank in Recommender Systems [4.058828240864671]
  Unbiased learning to rank algorithms are appealing candidates and have already been applied in many applications with single categorical labels. We propose a novel unbiased LTR algorithm to tackle the challenges, which innovatively models position bias in the pairwise fashion. Experiment results on public benchmark datasets and internal live traffic show the superior results of the proposed method for both categorical and continuous labels.
  arXiv  Detail & Related papers  (2021-11-25T06:04:59Z)
• A Pre-processing Method for Fairness in Ranking [0.0]
  We propose a fair ranking framework that evaluates the order of training data in a pairwise manner. We show that our method outperforms the existing methods in the trade-off between accuracy and fairness over real-world datasets.
  arXiv  Detail & Related papers  (2021-10-29T02:55:32Z)
• Distribution-free uncertainty quantification for classification under label shift [105.27463615756733]
  We focus on uncertainty quantification (UQ) for classification problems via two avenues. We first argue that label shift hurts UQ, by showing degradation in coverage and calibration. We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.
  arXiv  Detail & Related papers  (2021-03-04T20:51:03Z)
• Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning [8.780958735684958]
  Post-hoc multi-class calibration is a common approach for providing confidence estimates of deep neural network predictions. Recent work has shown that widely used scaling methods underestimate their calibration error. We propose a shared class-wise (sCW) calibration strategy, sharing one calibrator among similar classes.
  arXiv  Detail & Related papers  (2020-06-23T15:31:59Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.