Measurement and applications of position bias in a marketplace search engine

• URL: http://arxiv.org/abs/2206.11720v1
• Date: Thu, 23 Jun 2022 14:09:58 GMT
• Title: Measurement and applications of position bias in a marketplace search engine
• Authors: Richard Demsyn-Jones
• Abstract summary: Search engines intentionally influence user behavior by picking and ranking the list of results. This paper describes our efforts at Thumbtack to understand the impact of ranking. We include a novel discussion of how ranking bias may not only affect labels, but also model features.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Search engines intentionally influence user behavior by picking and ranking the list of results. Users engage with the highest results both because of their prominent placement and because they are typically the most relevant documents. Search engine ranking algorithms need to identify relevance while incorporating the influence of the search engine itself. This paper describes our efforts at Thumbtack to understand the impact of ranking, including the empirical results of a randomization program. In the context of a consumer marketplace we discuss practical details of model choice, experiment design, bias calculation, and machine learning model adaptation. We include a novel discussion of how ranking bias may not only affect labels, but also model features. The randomization program led to improved models, motivated internal scenario analysis, and enabled user-facing scenario tooling.

Related papers

• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs) Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings. By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)
• Learning Fair Ranking Policies via Differentiable Optimization of Ordered Weighted Averages [55.04219793298687]
  This paper shows how efficiently-solvable fair ranking models can be integrated into the training loop of Learning to Rank. In particular, this paper is the first to show how to backpropagate through constrained optimizations of OWA objectives, enabling their use in integrated prediction and decision models.
  arXiv  Detail & Related papers  (2024-02-07T20:53:53Z)
• Position bias in features [0.0]
  Document-specific historical click-through rates can be important features in a dynamic ranking system. This paper describes the properties of several such features, and tests them in controlled experiments.
  arXiv  Detail & Related papers  (2024-02-04T22:15:30Z)
• Whole Page Unbiased Learning to Rank [59.52040055543542]
  Unbiased Learning to Rank(ULTR) algorithms are proposed to learn an unbiased ranking model with biased click data. We propose a Bias Agnostic whole-page unbiased Learning to rank algorithm, named BAL, to automatically find the user behavior model. Experimental results on a real-world dataset verify the effectiveness of the BAL.
  arXiv  Detail & Related papers  (2022-10-19T16:53:08Z)
• Incorporating Vision Bias into Click Models for Image-oriented Search Engine [51.192784793764176]
  In this paper, we assume that vision bias exists in an image-oriented search engine as another crucial factor affecting the examination probability aside from position. We use regression-based EM algorithm to predict the vision bias given the visual features extracted from candidate documents.
  arXiv  Detail & Related papers  (2021-01-07T10:01:31Z)
• Characterizing Fairness Over the Set of Good Models Under Selective Labels [69.64662540443162]
  We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance. We provide tractable algorithms to compute the range of attainable group-level predictive disparities. We extend our framework to address the empirically relevant challenge of selectively labelled data.
  arXiv  Detail & Related papers  (2021-01-02T02:11:37Z)
• Choppy: Cut Transformer For Ranked List Truncation [92.58177016973421]
  Choppy is an assumption-free model based on the widely successful Transformer architecture. We show Choppy improves upon recent state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-04-26T00:52:49Z)
• A survey of bias in Machine Learning through the prism of Statistical Parity for the Adult Data Set [5.277804553312449]
  We show the importance of understanding how a bias can be introduced into automatic decisions. We first present a mathematical framework for the fair learning problem, specifically in the binary classification setting. We then propose to quantify the presence of bias by using the standard Disparate Impact index on the real and well-known Adult income data set.
  arXiv  Detail & Related papers  (2020-03-31T14:48:36Z)
• Handling Position Bias for Unbiased Learning to Rank in Hotels Search [0.951828574518325]
  We will investigate the importance of properly handling the position bias in an online test environment in Tripadvisor Hotels search. We propose an empirically effective method of handling the position bias that fully leverages the user action data. The online A/B test results show that this method leads to an improved search ranking model.
  arXiv  Detail & Related papers  (2020-02-28T03:48:42Z)
• Eliminating Search Intent Bias in Learning to Rank [0.32228025627337864]
  We study how differences in user search intent can influence click activities and determined that there exists a bias between user search intent and the relevance of the document relevance. We propose a search intent bias hypothesis that can be applied to most existing click models to improve their ability to learn unbiased relevance.
  arXiv  Detail & Related papers  (2020-02-08T17:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.