Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding

• URL: http://arxiv.org/abs/2404.14600v1
• Date: Mon, 22 Apr 2024 21:50:01 GMT
• Title: Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding
• Authors: Hansi Zeng, Chen Luo, Hamed Zamani,
• Abstract summary: This paper introduces PAG, a novel optimization and decoding approach that guides autoregressive generation of document identifiers. Experiments on MSMARCO and TREC Deep Learning Track data reveal that PAG outperforms the state-of-the-art generative retrieval model by a large margin.
• Score: 23.061797784952855
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces PAG-a novel optimization and decoding approach that guides autoregressive generation of document identifiers in generative retrieval models through simultaneous decoding. To this aim, PAG constructs a set-based and sequential identifier for each document. Motivated by the bag-of-words assumption in information retrieval, the set-based identifier is built on lexical tokens. The sequential identifier, on the other hand, is obtained via quantizing relevance-based representations of documents. Extensive experiments on MSMARCO and TREC Deep Learning Track data reveal that PAG outperforms the state-of-the-art generative retrieval model by a large margin (e.g., 15.6% MRR improvements on MS MARCO), while achieving 22x speed up in terms of query latency.

Related papers

• Learning More Effective Representations for Dense Retrieval through Deliberate Thinking Before Search [65.53881294642451]
  Deliberate Thinking based Dense Retriever (DEBATER) DEBATER enhances recent dense retrievers by enabling them to learn more effective document representations through a step-by-step thinking process. Experimental results show that DEBATER significantly outperforms existing methods across several retrieval benchmarks.
  arXiv  Detail & Related papers  (2025-02-18T15:56:34Z)
• Order-agnostic Identifier for Large Language Model-based Generative Recommendation [94.37662915542603]
  Items are assigned identifiers for Large Language Models (LLMs) to encode user history and generate the next item. Existing approaches leverage either token-sequence identifiers, representing items as discrete token sequences, or single-token identifiers, using ID or semantic embeddings. We propose SETRec, which leverages semantic tokenizers to obtain order-agnostic multi-dimensional tokens.
  arXiv  Detail & Related papers  (2025-02-15T15:25:38Z)
• DOGR: Leveraging Document-Oriented Contrastive Learning in Generative Retrieval [10.770281363775148]
  We propose a novel and general generative retrieval framework, namely Leveraging Document-Oriented Contrastive Learning in Generative Retrieval (DOGR) It adopts a two-stage learning strategy that captures the relationship between queries and documents comprehensively through direct interactions. Negative sampling methods and corresponding contrastive learning objectives are implemented to enhance the learning of semantic representations.
  arXiv  Detail & Related papers  (2025-02-11T03:25:42Z)
• Generative Retrieval Meets Multi-Graded Relevance [104.75244721442756]
  We introduce a framework called GRaded Generative Retrieval (GR$2$) GR$2$ focuses on two key components: ensuring relevant and distinct identifiers, and implementing multi-graded constrained contrastive training. Experiments on datasets with both multi-graded and binary relevance demonstrate the effectiveness of GR$2$.
  arXiv  Detail & Related papers  (2024-09-27T02:55:53Z)
• Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval [108.9772640854136]
  Generative retrieval uses differentiable search indexes to directly generate relevant document identifiers in response to a query. Recent studies have highlighted the potential of a strong generative retrieval model, trained with carefully crafted pre-training tasks, to enhance downstream retrieval tasks via fine-tuning. We introduce BootRet, a bootstrapped pre-training method for generative retrieval that dynamically adjusts document identifiers during pre-training to accommodate the continuing of the corpus.
  arXiv  Detail & Related papers  (2024-07-16T08:42:36Z)
• Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection [28.15184715270483]
  Large language models (LLMs) augmented with retrieval exhibit robust performance and extensive versatility. We propose a novel paradigm named Sparse RAG, which seeks to cut costs through sparsity. Sparse RAG encodes retrieved documents in parallel, which eliminates latency introduced by long-range attention of retrieved documents.
  arXiv  Detail & Related papers  (2024-05-25T11:10:04Z)
• ASI++: Towards Distributionally Balanced End-to-End Generative Retrieval [29.65717446547002]
  ASI++ is a novel fully end-to-end generative retrieval method. It aims to simultaneously learn balanced ID assignments and improve retrieval performance.
  arXiv  Detail & Related papers  (2024-05-23T07:54:57Z)
• Summarization-Based Document IDs for Generative Retrieval with Language Models [65.11811787587403]
  We introduce summarization-based document IDs, in which each document's ID is composed of an extractive summary or abstractive keyphrases. We show that using ACID improves top-10 and top-20 recall by 15.6% and 14.4% (relative) respectively. We also observed that extractive IDs outperformed abstractive IDs on Wikipedia articles in NQ but not the snippets in MSMARCO.
  arXiv  Detail & Related papers  (2023-11-14T23:28:36Z)
• Autoregressive Search Engines: Generating Substrings as Document Identifiers [53.0729058170278]
  Autoregressive language models are emerging as the de-facto standard for generating answers. Previous work has explored ways to partition the search space into hierarchical structures. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers.
  arXiv  Detail & Related papers  (2022-04-22T10:45:01Z)
• Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval [11.465218502487959]
  We design a method to mimic the queries on each of the documents by an iterative clustering process. We also optimize the matching function with a two-step score calculation procedure. Experimental results on several popular ranking and QA datasets show that our model can achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2021-05-08T05:28:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.