Beyond Pairwise Learning-To-Rank At Airbnb

• URL: http://arxiv.org/abs/2505.09795v3
• Date: Sun, 01 Jun 2025 20:17:37 GMT
• Title: Beyond Pairwise Learning-To-Rank At Airbnb
• Authors: Malay Haldar, Daochen Zha, Huiji Gao, Liwei He, Sanjeev Katariya,
• Abstract summary: We call this limitation the SAT theorem for ranking algorithms.<n>Our current work at Airbnb provides an answer, with a working solution deployed at scale.
• Score: 25.51587160196683
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There are three fundamental asks from a ranking algorithm: it should scale to handle a large number of items, sort items accurately by their utility, and impose a total order on the items for logical consistency. But here's the catch-no algorithm can achieve all three at the same time. We call this limitation the SAT theorem for ranking algorithms. Given the dilemma, how can we design a practical system that meets user needs? Our current work at Airbnb provides an answer, with a working solution deployed at scale. We start with pairwise learning-to-rank (LTR) models-the bedrock of search ranking tech stacks today. They scale linearly with the number of items ranked and perform strongly on metrics like NDCG by learning from pairwise comparisons. They are at a sweet spot of performance vs. cost, making them an ideal choice for several industrial applications. However, they have a drawback-by ignoring interactions between items, they compromise on accuracy. To improve accuracy, we create a "true" pairwise LTR model-one that captures interactions between items during pairwise comparisons. But accuracy comes at the expense of scalability and total order, and we discuss strategies to counter these challenges. For greater accuracy, we take each item in the search result, and compare it against the rest of the items along two dimensions: (1) Superiority: How strongly do searchers prefer the given item over the remaining ones? (2) Similarity: How similar is the given item to all the other items? This forms the basis of our "all-pairwise" LTR framework, which factors in interactions across all items at once. Looking at items on the search result page all together-superiority and similarity combined-gives us a deeper understanding of what searchers truly want. We quantify the resulting improvements in searcher experience through offline and online experiments at Airbnb.

Related papers

• DeepShop: A Benchmark for Deep Research Shopping Agents [70.03744154560717]
  DeepShop is a benchmark designed to evaluate web agents in complex and realistic online shopping environments.<n>We generate diverse queries across five popular online shopping domains.<n>We propose an automated evaluation framework that assesses agent performance in terms of fine-grained aspects.
  arXiv  Detail & Related papers  (2025-06-03T13:08:17Z)
• Scalable Private Partition Selection via Adaptive Weighting [66.09199304818928]
  In a private set union, users hold subsets of items from an unbounded universe.<n>The goal is to output as many items as possible from the union of the users' sets while maintaining user-level differential privacy.<n>We propose an algorithm for this problem, MaximumDegree (MAD), which adaptively reroutes weight from items with weight far above the threshold needed for privacy to items with smaller weight.
  arXiv  Detail & Related papers  (2025-02-13T01:27:11Z)
• Relevance Filtering for Embedding-based Retrieval [46.851594313019895]
  In embedding-based retrieval, Approximate Nearest Neighbor (ANN) search enables efficient retrieval of similar items from large-scale datasets. This paper introduces a novel relevance filtering component (called "Cosine Adapter") for embedding-based retrieval to address this challenge. We are able to significantly increase the precision of the retrieved set, at the expense of a small loss of recall.
  arXiv  Detail & Related papers  (2024-08-09T06:21:20Z)
• Active Preference Learning for Ordering Items In- and Out-of-sample [7.0774164818430565]
  actively sampling item pairs can reduce the number of annotations necessary for learning an accurate ordering. Many algorithms ignore shared structure between items. It is also common to disregard how noise in comparisons varies between item pairs.
  arXiv  Detail & Related papers  (2024-05-05T21:44:03Z)
• List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented Generation [80.12531449946655]
  We propose a Reranking-Truncation joint model (GenRT) that can perform the two tasks concurrently. GenRT integrates reranking and truncation via generative paradigm based on encoder-decoder architecture. Our method achieves SOTA performance on both reranking and truncation tasks for web search and retrieval-augmented LLMs.
  arXiv  Detail & Related papers  (2024-02-05T06:52:53Z)
• Beyond Two-Tower Matching: Learning Sparse Retrievable Cross-Interactions for Recommendation [80.19762472699814]
  Two-tower models are a prevalent matching framework for recommendation, which have been widely deployed in industrial applications. It suffers two main challenges, including limited feature interaction capability and reduced accuracy in online serving. We propose a new matching paradigm named SparCode, which supports not only sophisticated feature interactions but also efficient retrieval.
  arXiv  Detail & Related papers  (2023-11-30T03:13:36Z)
• Unsupervised Contrast-Consistent Ranking with Language Models [24.696017700382665]
  Language models contain ranking-based knowledge and are powerful solvers of in-context ranking tasks. We compare pairwise, pointwise and listwise prompting techniques to elicit a language model's ranking knowledge. We find that even with careful calibration and constrained decoding, prompting-based techniques may not always be self-consistent in the rankings they produce.
  arXiv  Detail & Related papers  (2023-09-13T14:36:26Z)
• Online Learning of Optimally Diverse Rankings [63.62764375279861]
  We propose an algorithm that efficiently learns the optimal list based on users' feedback only. We show that after $T$ queries, the regret of LDR scales as $O((N-L)log(T))$ where $N$ is the number of all items.
  arXiv  Detail & Related papers  (2021-09-13T12:13:20Z)
• IRLI: Iterative Re-partitioning for Learning to Index [104.72641345738425]
  Methods have to trade between obtaining high accuracy while maintaining load balance and scalability in distributed settings. We propose a novel approach called IRLI, which iteratively partitions the items by learning the relevant buckets directly from the query-item relevance data. We mathematically show that IRLI retrieves the correct item with high probability under very natural assumptions and provides superior load balancing.
  arXiv  Detail & Related papers  (2021-03-17T23:13:25Z)
• Controlling Fairness and Bias in Dynamic Learning-to-Rank [31.41843594914603]
  We propose a learning algorithm that ensures notions of amortized group fairness, while simultaneously learning the ranking function from implicit feedback data. The algorithm takes the form of a controller that integrates unbiased estimators for both fairness and utility. In addition to its rigorous theoretical foundation and convergence guarantees, we find empirically that the algorithm is highly practical and robust.
  arXiv  Detail & Related papers  (2020-05-29T17:57:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.