The Surprising Effectiveness of Rankers Trained on Expanded Queries

• URL: http://arxiv.org/abs/2404.02587v2
• Date: Wed, 12 Jun 2024 09:34:43 GMT
• Title: The Surprising Effectiveness of Rankers Trained on Expanded Queries
• Authors: Abhijit Anand, Venktesh V, Vinay Setty, Avishek Anand,
• Abstract summary: We improve the ranking performance of hard or difficult queries without compromising the performance of other queries. We combine relevance scores from the specialized ranker and the base ranker, along with a query performance score estimated for each query. In our experiments on the DL-Hard dataset, we find that a principled query performance based scoring method offers a significant improvement of up to 25% on the passage ranking task.
• Score: 4.874071145951159
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: An important problem in text-ranking systems is handling the hard queries that form the tail end of the query distribution. The difficulty may arise due to the presence of uncommon, underspecified, or incomplete queries. In this work, we improve the ranking performance of hard or difficult queries without compromising the performance of other queries. Firstly, we do LLM based query enrichment for training queries using relevant documents. Next, a specialized ranker is fine-tuned only on the enriched hard queries instead of the original queries. We combine the relevance scores from the specialized ranker and the base ranker, along with a query performance score estimated for each query. Our approach departs from existing methods that usually employ a single ranker for all queries, which is biased towards easy queries, which form the majority of the query distribution. In our extensive experiments on the DL-Hard dataset, we find that a principled query performance based scoring method using base and specialized ranker offers a significant improvement of up to 25% on the passage ranking task and up to 48.4% on the document ranking task when compared to the baseline performance of using original queries, even outperforming SOTA model.

Related papers

• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  Many complex real-world queries require in-depth reasoning to identify relevant documents. We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. brightbenchmark is constructed from the 1,398 real-world queries collected from diverse domains.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)
• Database-Augmented Query Representation for Information Retrieval [59.57065228857247]
  We present a novel retrieval framework called Database-Augmented Query representation (DAQu) DAQu augments the original query with various (query-related) metadata across multiple tables. We validate DAQu in diverse retrieval scenarios that can incorporate metadata from the relational database.
  arXiv  Detail & Related papers  (2024-06-23T05:02:21Z)
• Context Aware Query Rewriting for Text Rankers using LLM [5.164642900490078]
  We analyze the utility of large-language models for improved query rewriting for text ranking tasks. We adopt a simple, yet surprisingly effective, approach called context aware query rewriting (CAR) We find that fine-tuning a ranker using re-written queries offers a significant improvement of up to 33% on the passage ranking task and up to 28% on the document ranking task.
  arXiv  Detail & Related papers  (2023-08-31T14:19:50Z)
• CAPSTONE: Curriculum Sampling for Dense Retrieval with Document Expansion [68.19934563919192]
  We propose a curriculum sampling strategy that utilizes pseudo queries during training and progressively enhances the relevance between the generated query and the real query. Experimental results on both in-domain and out-of-domain datasets demonstrate that our approach outperforms previous dense retrieval models.
  arXiv  Detail & Related papers  (2022-12-18T15:57:46Z)
• Searching for Better Database Queries in the Outputs of Semantic Parsers [16.221439565760058]
  In this paper, we consider the case when, at the test time, the system has access to an external criterion that evaluates the generated queries. The criterion can vary from checking that a query executes without errors to verifying the query on a set of tests. We apply our approach to the state-of-the-art semantics and report that it allows us to find many queries passing all the tests on different datasets.
  arXiv  Detail & Related papers  (2022-10-13T17:20:45Z)
• Counterfactual Learning To Rank for Utility-Maximizing Query Autocompletion [40.31426350180036]
  We propose a new approach that explicitly optimize the query suggestions for downstream retrieval performance. We formulate this as a problem of ranking a set of rankings, where each query suggestion is represented by the downstream item ranking it produces. We then present a learning method that ranks query suggestions by the quality of their item rankings.
  arXiv  Detail & Related papers  (2022-04-22T21:40:51Z)
• Graph Enhanced BERT for Query Understanding [55.90334539898102]
  query understanding plays a key role in exploring users' search intents and facilitating users to locate their most desired information. In recent years, pre-trained language models (PLMs) have advanced various natural language processing tasks. We propose a novel graph-enhanced pre-training framework, GE-BERT, which can leverage both query content and the query graph.
  arXiv  Detail & Related papers  (2022-04-03T16:50:30Z)
• Exposing Query Identification for Search Transparency [69.06545074617685]
  We explore the feasibility of approximate exposing query identification (EQI) as a retrieval task by reversing the role of queries and documents in two classes of search systems. We derive an evaluation metric to measure the quality of a ranking of exposing queries, as well as conducting an empirical analysis focusing on various practical aspects of approximate EQI.
  arXiv  Detail & Related papers  (2021-10-14T20:19:27Z)
• Leveraging semantically similar queries for ranking via combining representations [20.79800117378761]
  In data-scarce settings, the amount of labeled data available for a particular query can lead to a highly variable and ineffective ranking function. One way to mitigate the effect of the small amount of data is to leverage information from semantically similar queries. We describe and explore this phenomenon in the context of the bias-variance trade off and apply it to the data-scarce settings of a Bing navigational graph and the Drosophila larva connectome.
  arXiv  Detail & Related papers  (2021-06-23T18:36:20Z)
• Surprise: Result List Truncation via Extreme Value Theory [92.5817701697342]
  We propose a statistical method that produces interpretable and calibrated relevance scores at query time using nothing more than the ranked scores. We demonstrate its effectiveness on the result list truncation task across image, text, and IR datasets.
  arXiv  Detail & Related papers  (2020-10-19T19:15:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.