Analysis of Multivariate Scoring Functions for Automatic Unbiased Learning to Rank

• URL: http://arxiv.org/abs/2008.09061v1
• Date: Thu, 20 Aug 2020 16:31:59 GMT
• Title: Analysis of Multivariate Scoring Functions for Automatic Unbiased Learning to Rank
• Authors: Tao Yang, Shikai Fang, Shibo Li, Yulan Wang, Qingyao Ai
• Abstract summary: AutoULTR algorithms that jointly learn user bias models (i.e., propensity models) with unbiased rankers have received a lot of attention due to their superior performance and low deployment cost in practice. Recent advances in context-aware learning-to-rank models have shown that multivariate scoring functions, which read multiple documents together and predict their ranking scores jointly, are more powerful than uni-variate ranking functions in ranking tasks with human-annotated relevance labels. Our experiments with synthetic clicks on two large-scale benchmark datasets show that AutoULTR models with permutation-invariant multivariate scoring functions significantly outperform
• Score: 14.827143632277274
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Leveraging biased click data for optimizing learning to rank systems has been a popular approach in information retrieval. Because click data is often noisy and biased, a variety of methods have been proposed to construct unbiased learning to rank (ULTR) algorithms for the learning of unbiased ranking models. Among them, automatic unbiased learning to rank (AutoULTR) algorithms that jointly learn user bias models (i.e., propensity models) with unbiased rankers have received a lot of attention due to their superior performance and low deployment cost in practice. Despite their differences in theories and algorithm design, existing studies on ULTR usually use uni-variate ranking functions to score each document or result independently. On the other hand, recent advances in context-aware learning-to-rank models have shown that multivariate scoring functions, which read multiple documents together and predict their ranking scores jointly, are more powerful than uni-variate ranking functions in ranking tasks with human-annotated relevance labels. Whether such superior performance would hold in ULTR with noisy data, however, is mostly unknown. In this paper, we investigate existing multivariate scoring functions and AutoULTR algorithms in theory and prove that permutation invariance is a crucial factor that determines whether a context-aware learning-to-rank model could be applied to existing AutoULTR framework. Our experiments with synthetic clicks on two large-scale benchmark datasets show that AutoULTR models with permutation-invariant multivariate scoring functions significantly outperform those with uni-variate scoring functions and permutation-variant multivariate scoring functions.

Related papers

• Contextual Dual Learning Algorithm with Listwise Distillation for Unbiased Learning to Rank [26.69630281310365]
  Unbiased Learning to Rank (ULTR) aims to leverage biased implicit user feedback (e.g., click) to optimize an unbiased ranking model. We propose a Contextual Dual Learning Algorithm with Listwise Distillation (CDLA-LD) to address both position bias and contextual bias.
  arXiv  Detail & Related papers  (2024-08-19T09:13:52Z)
• Learning Fair Ranking Policies via Differentiable Optimization of Ordered Weighted Averages [55.04219793298687]
  This paper shows how efficiently-solvable fair ranking models can be integrated into the training loop of Learning to Rank. In particular, this paper is the first to show how to backpropagate through constrained optimizations of OWA objectives, enabling their use in integrated prediction and decision models.
  arXiv  Detail & Related papers  (2024-02-07T20:53:53Z)
• Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment [54.179859639868646]
  We propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories.
  arXiv  Detail & Related papers  (2023-07-27T07:42:44Z)
• Deep Negative Correlation Classification [82.45045814842595]
  Existing deep ensemble methods naively train many different models and then aggregate their predictions. We propose deep negative correlation classification (DNCC) DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
  arXiv  Detail & Related papers  (2022-12-14T07:35:20Z)
• piRank: A Probabilistic Intent Based Ranking Framework for Facebook Search [0.07614628596146598]
  We propose a probabilistic intent based ranking framework (short for piRank) to address various ranking issues for different query intents. We conducted extensive experiments and studies on top of Facebook search engine system and validated the effectiveness of this new ranking architecture.
  arXiv  Detail & Related papers  (2022-03-27T18:12:56Z)
• Boosting the Learning for Ranking Patterns [6.142272540492935]
  This paper formulates the problem of learning pattern ranking functions as a multi-criteria decision making problem. Our approach aggregates different interestingness measures into a single weighted linear ranking function, using an interactive learning procedure. Experiments conducted on well-known datasets show that our approach significantly reduces the running time and returns precise pattern ranking.
  arXiv  Detail & Related papers  (2022-03-05T10:22:44Z)
• ULTRA: An Unbiased Learning To Rank Algorithm Toolbox [13.296248894004652]
  In this paper, we describe the general framework of unbiased learning to rank (ULTR) We also briefly describe the algorithms in ULTRA, detailed the structure, and pipeline of the toolbox. Our toolbox is an important resource for researchers to conduct experiments on ULTR algorithms with different configurations as well as testing their own algorithms with the supported features.
  arXiv  Detail & Related papers  (2021-08-11T07:26:59Z)
• Learning High-Order Interactions via Targeted Pattern Search [0.6198237241838558]
  We present a novel algorithm, Learning high-order Interactions via targeted Pattern Search (LIPS) LIPS selects interaction terms of varying order to include in a Logistic Regression model. We prove its wide applicability to real-life research scenarios, showing that it outperforms a benchmark state-of-the-art algorithm.
  arXiv  Detail & Related papers  (2021-02-23T11:13:22Z)
• Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank [62.28965622396868]
  We introduce the Generalization and generalization (GENSPEC) algorithm, a robust feature-based counterfactual Learning to Rank method. Our results show that GENSPEC leads to optimal performance on queries with sufficient click data, while having robust behavior on queries with little or noisy data.
  arXiv  Detail & Related papers  (2021-02-11T13:17:26Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• PiRank: Learning To Rank via Differentiable Sorting [85.28916333414145]
  We propose PiRank, a new class of differentiable surrogates for ranking. We show that PiRank exactly recovers the desired metrics in the limit of zero temperature.
  arXiv  Detail & Related papers  (2020-12-12T05:07:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.