Unbiased Learning to Rank with Query-Level Click Propensity Estimation: Beyond Pointwise Observation and Relevance

• URL: http://arxiv.org/abs/2502.11414v2
• Date: Tue, 18 Feb 2025 07:04:29 GMT
• Title: Unbiased Learning to Rank with Query-Level Click Propensity Estimation: Beyond Pointwise Observation and Relevance
• Authors: Lulu Yu, Keping Bi, Jiafeng Guo, Shihao Liu, Dawei Yin, Xueqi Cheng,
• Abstract summary: In real-world scenarios, users often click only one or two results after examining multiple relevant options. We propose a query-level click propensity model to capture the probability that users will click on different result lists. Our method introduces a Dual Inverse Propensity Weighting mechanism to address both relevance saturation and position bias.
• Score: 74.43264459255121
• License:
• Abstract: Most existing unbiased learning-to-rank (ULTR) approaches are based on the user examination hypothesis, which assumes that users will click a result only if it is both relevant and observed (typically modeled by position). However, in real-world scenarios, users often click only one or two results after examining multiple relevant options, due to limited patience or because their information needs have already been satisfied. Motivated by this, we propose a query-level click propensity model to capture the probability that users will click on different result lists, allowing for non-zero probabilities that users may not click on an observed relevant result. We hypothesize that this propensity increases when more potentially relevant results are present, and refer to this user behavior as relevance saturation bias. Our method introduces a Dual Inverse Propensity Weighting (DualIPW) mechanism -- combining query-level and position-level IPW -- to address both relevance saturation and position bias. Through theoretical derivation, we prove that DualIPW can learn an unbiased ranking model. Experiments on the real-world Baidu-ULTR dataset demonstrate that our approach significantly outperforms state-of-the-art ULTR baselines. The code and dataset information can be found at https://github.com/Trustworthy-Information-Access/DualIPW.

Related papers

• Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset [48.708591046906896]
  Unbiased learning-to-rank (ULTR) is a well-established framework for learning from user clicks. We revisit and extend the available experiments on the Baidu-ULTR dataset. We find that standard unbiased learning-to-rank techniques robustly improve click predictions but struggle to consistently improve ranking performance.
  arXiv  Detail & Related papers  (2024-04-03T08:00:46Z)
• An Offline Metric for the Debiasedness of Click Models [52.25681483524383]
  Click models are a common method for extracting information from user clicks. Recent work shows that the current evaluation practices in the community fail to guarantee that a well-performing click model generalizes well to downstream tasks. We introduce the concept of debiasedness in click modeling and derive a metric for measuring it.
  arXiv  Detail & Related papers  (2023-04-19T10:59:34Z)
• Whole Page Unbiased Learning to Rank [59.52040055543542]
  Unbiased Learning to Rank(ULTR) algorithms are proposed to learn an unbiased ranking model with biased click data. We propose a Bias Agnostic whole-page unbiased Learning to rank algorithm, named BAL, to automatically find the user behavior model. Experimental results on a real-world dataset verify the effectiveness of the BAL.
  arXiv  Detail & Related papers  (2022-10-19T16:53:08Z)
• A Large Scale Search Dataset for Unbiased Learning to Rank [51.97967284268577]
  We introduce the Baidu-ULTR dataset for unbiased learning to rank. It involves randomly sampled 1.2 billion searching sessions and 7,008 expert annotated queries. It provides: (1) the original semantic feature and a pre-trained language model for easy usage; (2) sufficient display information such as position, displayed height, and displayed abstract; and (3) rich user feedback on search result pages (SERPs) like dwelling time.
  arXiv  Detail & Related papers  (2022-07-07T02:37:25Z)
• Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank [29.934700345584726]
  Unbiased learning to rank aims to train an unbiased ranking model from biased user click logs. Most of the current ULTR methods are based on the examination hypothesis (EH), which assumes that the click probability can be factorized into two scalar functions. We propose a vector-based EH and formulate the click probability as a dot product of two vector functions.
  arXiv  Detail & Related papers  (2022-06-03T17:23:25Z)
• Cross Pairwise Ranking for Unbiased Item Recommendation [57.71258289870123]
  We develop a new learning paradigm named Cross Pairwise Ranking (CPR) CPR achieves unbiased recommendation without knowing the exposure mechanism. We prove in theory that this way offsets the influence of user/item propensity on the learning.
  arXiv  Detail & Related papers  (2022-04-26T09:20:27Z)
• Handling Position Bias for Unbiased Learning to Rank in Hotels Search [0.951828574518325]
  We will investigate the importance of properly handling the position bias in an online test environment in Tripadvisor Hotels search. We propose an empirically effective method of handling the position bias that fully leverages the user action data. The online A/B test results show that this method leads to an improved search ranking model.
  arXiv  Detail & Related papers  (2020-02-28T03:48:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.