Unleashing the Power of LLMs in Dense Retrieval with Query Likelihood Modeling

• URL: http://arxiv.org/abs/2504.05216v1
• Date: Mon, 07 Apr 2025 16:03:59 GMT
• Title: Unleashing the Power of LLMs in Dense Retrieval with Query Likelihood Modeling
• Authors: Hengran Zhang, Keping Bi, Jiafeng Guo, Xiaojie Sun, Shihao Liu, Daiting Shi, Dawei Yin, Xueqi Cheng,
• Abstract summary: Large language models (LLMs) have shown compelling semantic understanding capabilities.<n>Dense retrieval is a crucial task in Information Retrieval (IR) and is the foundation for downstream tasks as re-ranking.<n>We introduce an auxiliary task of QL estimation to yield a better backbone for contrast learning a discriminative retriever.
• Score: 69.84963245729826
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dense retrieval is a crucial task in Information Retrieval (IR) and is the foundation for downstream tasks such as re-ranking. Recently, large language models (LLMs) have shown compelling semantic understanding capabilities and are appealing to researchers studying dense retrieval. LLMs, as decoder-style generative models, are competent at language generation while falling short on modeling global information due to the lack of attention to tokens afterward. Inspired by the classical word-based language modeling approach for IR, i.e., the query likelihood (QL) model, we seek to sufficiently utilize LLMs' generative ability by QL maximization. However, instead of ranking documents with QL estimation, we introduce an auxiliary task of QL maximization to yield a better backbone for contrastively learning a discriminative retriever. We name our model as LLM-QL. To condense global document semantics to a single vector during QL modeling, LLM-QL has two major components, Attention Stop (AS) and Input Corruption (IC). AS stops the attention of predictive tokens to previous tokens until the ending token of the document. IC masks a portion of tokens in the input documents during prediction. Experiments on MSMARCO show that LLM-QL can achieve significantly better performance than other LLM-based retrievers and using QL estimated by LLM-QL for ranking outperforms word-based QL by a large margin.

Related papers

• LLM-QE: Improving Query Expansion by Aligning Large Language Models with Ranking Preferences [21.777817032607405]
  This paper introduces LLM-QE, a novel approach that leverages Large Language Models (LLMs) to generate document-based query expansions.<n>Experiments on the zero-shot dense retrieval model, Contriever, demonstrate the effectiveness of LLM-QE, achieving an improvement of over 8%.<n>LLM-QE also improves the training process of dense retrievers, achieving a more than 5% improvement after fine-tuning.
  arXiv  Detail & Related papers  (2025-02-24T11:15:41Z)
• Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision [50.45597801390757]
  Instruct-LF is a goal-oriented latent factor discovery system. It integrates instruction-following ability with statistical models to handle noisy datasets.
  arXiv  Detail & Related papers  (2025-02-21T02:03:08Z)
• NAT-NL2GQL: A Novel Multi-Agent Framework for Translating Natural Language to Graph Query Language [13.661054027428868]
  We propose NAT-NL2GQL, a novel framework for translating natural language to graph query language. Our framework consists of three synergistic agents: the Preprocessor agent, the Generator agent, and the Refiner agent. Given the scarcity of high-quality open-source NL2GQL datasets based on nGQL syntax, we developed StockGQL, a dataset constructed from a financial market graph database.
  arXiv  Detail & Related papers  (2024-12-11T04:14:09Z)
• Less is More: Making Smaller Language Models Competent Subgraph Retrievers for Multi-hop KGQA [51.3033125256716]
  We model the subgraph retrieval task as a conditional generation task handled by small language models. Our base generative subgraph retrieval model, consisting of only 220M parameters, competitive retrieval performance compared to state-of-the-art models. Our largest 3B model, when plugged with an LLM reader, sets new SOTA end-to-end performance on both the WebQSP and CWQ benchmarks.
  arXiv  Detail & Related papers  (2024-10-08T15:22:36Z)
• Assessing SPARQL capabilities of Large Language Models [0.0]
  We focus on measuring out-of-the box capabilities of Large Language Models to work with SPARQL.<n>We implement benchmarking tasks in the LLM-KG-Bench framework for automated execution and evaluation.<n>Our findings indicate that working with SPARQL SELECT queries is still challenging for LLMs.
  arXiv  Detail & Related papers  (2024-09-09T08:29:39Z)
• LLM-PQA: LLM-enhanced Prediction Query Answering [7.346989832385652]
  This paper introduces LLM-PQA, a novel tool that addresses prediction queries formulated in natural language. This integration provides users with access to a vast spectrum of heterogeneous data and diverse ML models, facilitating dynamic prediction query answering.
  arXiv  Detail & Related papers  (2024-09-02T10:20:35Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• LLatrieval: LLM-Verified Retrieval for Verifiable Generation [67.93134176912477]
  Verifiable generation aims to let the large language model (LLM) generate text with supporting documents. We propose LLatrieval (Large Language Model Verified Retrieval), where the LLM updates the retrieval result until it verifies that the retrieved documents can sufficiently support answering the question. Experiments show that LLatrieval significantly outperforms extensive baselines and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2023-11-14T01:38:02Z)
• Description-Based Text Similarity [59.552704474862004]
  We identify the need to search for texts based on abstract descriptions of their content. We propose an alternative model that significantly improves when used in standard nearest neighbor search.
  arXiv  Detail & Related papers  (2023-05-21T17:14:31Z)
• Large Language Models are Strong Zero-Shot Retriever [89.16756291653371]
  We propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios. Our method, the Language language model as Retriever (LameR), is built upon no other neural models but an LLM.
  arXiv  Detail & Related papers  (2023-04-27T14:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.