Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding

• URL: http://arxiv.org/abs/2404.14600v1
• Date: Mon, 22 Apr 2024 21:50:01 GMT
• Title: Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding
• Authors: Hansi Zeng, Chen Luo, Hamed Zamani,
• Abstract summary: This paper introduces PAG, a novel optimization and decoding approach that guides autoregressive generation of document identifiers. Experiments on MSMARCO and TREC Deep Learning Track data reveal that PAG outperforms the state-of-the-art generative retrieval model by a large margin.
• Score: 23.061797784952855
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces PAG-a novel optimization and decoding approach that guides autoregressive generation of document identifiers in generative retrieval models through simultaneous decoding. To this aim, PAG constructs a set-based and sequential identifier for each document. Motivated by the bag-of-words assumption in information retrieval, the set-based identifier is built on lexical tokens. The sequential identifier, on the other hand, is obtained via quantizing relevance-based representations of documents. Extensive experiments on MSMARCO and TREC Deep Learning Track data reveal that PAG outperforms the state-of-the-art generative retrieval model by a large margin (e.g., 15.6% MRR improvements on MS MARCO), while achieving 22x speed up in terms of query latency.

Related papers

• Generative Retrieval Meets Multi-Graded Relevance [104.75244721442756]
  We introduce a framework called GRaded Generative Retrieval (GR$2$) GR$2$ focuses on two key components: ensuring relevant and distinct identifiers, and implementing multi-graded constrained contrastive training. Experiments on datasets with both multi-graded and binary relevance demonstrate the effectiveness of GR$2$.
  arXiv  Detail & Related papers  (2024-09-27T02:55:53Z)
• Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval [108.9772640854136]
  Generative retrieval uses differentiable search indexes to directly generate relevant document identifiers in response to a query. Recent studies have highlighted the potential of a strong generative retrieval model, trained with carefully crafted pre-training tasks, to enhance downstream retrieval tasks via fine-tuning. We introduce BootRet, a bootstrapped pre-training method for generative retrieval that dynamically adjusts document identifiers during pre-training to accommodate the continuing of the corpus.
  arXiv  Detail & Related papers  (2024-07-16T08:42:36Z)
• Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection [28.15184715270483]
  Large language models (LLMs) augmented with retrieval exhibit robust performance and extensive versatility. We propose a novel paradigm named Sparse RAG, which seeks to cut costs through sparsity. Sparse RAG encodes retrieved documents in parallel, which eliminates latency introduced by long-range attention of retrieved documents.
  arXiv  Detail & Related papers  (2024-05-25T11:10:04Z)
• ASI++: Towards Distributionally Balanced End-to-End Generative Retrieval [29.65717446547002]
  ASI++ is a novel fully end-to-end generative retrieval method. It aims to simultaneously learn balanced ID assignments and improve retrieval performance.
  arXiv  Detail & Related papers  (2024-05-23T07:54:57Z)
• Generative Retrieval as Multi-Vector Dense Retrieval [71.75503049199897]
  Generative retrieval generates identifiers of relevant documents in an end-to-end manner. Prior work has demonstrated that generative retrieval with atomic identifiers is equivalent to single-vector dense retrieval. We show that generative retrieval and multi-vector dense retrieval share the same framework for measuring the relevance to a query of a document.
  arXiv  Detail & Related papers  (2024-03-31T13:29:43Z)
• CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks [20.390672895839757]
  Retrieval-augmented generation (RAG) has emerged as a popular solution to enhance factual accuracy. Traditional retrieval modules often rely on large document index and disconnect with generative tasks. We propose textbfCorpusLM, a unified language model that integrates generative retrieval, closed-book generation, and RAG.
  arXiv  Detail & Related papers  (2024-02-02T06:44:22Z)
• Summarization-Based Document IDs for Generative Retrieval with Language Models [65.11811787587403]
  We introduce summarization-based document IDs, in which each document's ID is composed of an extractive summary or abstractive keyphrases. We show that using ACID improves top-10 and top-20 recall by 15.6% and 14.4% (relative) respectively. We also observed that extractive IDs outperformed abstractive IDs on Wikipedia articles in NQ but not the snippets in MSMARCO.
  arXiv  Detail & Related papers  (2023-11-14T23:28:36Z)
• Learning to Rank in Generative Retrieval [62.91492903161522]
  Generative retrieval aims to generate identifier strings of relevant passages as the retrieval target. We propose a learning-to-rank framework for generative retrieval, dubbed LTRGR. This framework only requires an additional learning-to-rank training phase to enhance current generative retrieval systems.
  arXiv  Detail & Related papers  (2023-06-27T05:48:14Z)
• Autoregressive Search Engines: Generating Substrings as Document Identifiers [53.0729058170278]
  Autoregressive language models are emerging as the de-facto standard for generating answers. Previous work has explored ways to partition the search space into hierarchical structures. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers.
  arXiv  Detail & Related papers  (2022-04-22T10:45:01Z)
• Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval [11.465218502487959]
  We design a method to mimic the queries on each of the documents by an iterative clustering process. We also optimize the matching function with a two-step score calculation procedure. Experimental results on several popular ranking and QA datasets show that our model can achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2021-05-08T05:28:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.