NAIL: Lexical Retrieval Indices with Efficient Non-Autoregressive Decoders

• URL: http://arxiv.org/abs/2305.14499v2
• Date: Mon, 23 Oct 2023 14:46:34 GMT
• Title: NAIL: Lexical Retrieval Indices with Efficient Non-Autoregressive Decoders
• Authors: Livio Baldini Soares, Daniel Gillick, Jeremy R. Cole, Tom Kwiatkowski
• Abstract summary: We present a method of capturing up to 86% of the gains of a Transformer cross-attention model with a lexicalized scoring function. We introduce NAIL as a model architecture that is compatible with recent encoder-decoder and decoder-only large language models.
• Score: 9.400555345874988
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural document rerankers are extremely effective in terms of accuracy. However, the best models require dedicated hardware for serving, which is costly and often not feasible. To avoid this serving-time requirement, we present a method of capturing up to 86% of the gains of a Transformer cross-attention model with a lexicalized scoring function that only requires 10-6% of the Transformer's FLOPs per document and can be served using commodity CPUs. When combined with a BM25 retriever, this approach matches the quality of a state-of-the art dual encoder retriever, that still requires an accelerator for query encoding. We introduce NAIL (Non-Autoregressive Indexing with Language models) as a model architecture that is compatible with recent encoder-decoder and decoder-only large language models, such as T5, GPT-3 and PaLM. This model architecture can leverage existing pre-trained checkpoints and can be fine-tuned for efficiently constructing document representations that do not require neural processing of queries.

Related papers

• Are Decoder-Only Large Language Models the Silver Bullet for Code Search? [32.338318300589776]
  This study presents the first systematic exploration of decoder-only large language models for code search. We evaluate nine state-of-the-art decoder-only models using two fine-tuning methods, two datasets, and three model sizes. Our findings reveal that fine-tuned CodeGemma significantly outperforms encoder-only models like UniXcoder.
  arXiv  Detail & Related papers  (2024-10-29T17:05:25Z)
• Shallow Cross-Encoders for Low-Latency Retrieval [69.06104373460597]
  Cross-Encoders based on large transformer models (such as BERT or T5) are computationally expensive and allow for scoring only a small number of documents within a reasonably small latency window. We show that weaker shallow transformer models (i.e., transformers with a limited number of layers) actually perform better than full-scale models when constrained to these practical low-latency settings.
  arXiv  Detail & Related papers  (2024-03-29T15:07:21Z)
• LOCOST: State-Space Models for Long Document Abstractive Summarization [76.31514220737272]
  We propose LOCOST: an encoder-decoder architecture based on state-space models for conditional text generation with long context inputs. With a computational complexity of $O(L log L)$, this architecture can handle significantly longer sequences than state-of-the-art models that are based on sparse attention patterns.
  arXiv  Detail & Related papers  (2024-01-31T15:33:37Z)
• Legal-HNet: Mixing Legal Long-Context Tokens with Hartley Transform [0.0]
  We introduce a new hybrid Seq2Seq architecture, a no-attention-based encoder connected with an attention-based decoder, which performs quite well on existing summarization tasks. This not only makes training models from scratch accessible to more people, but also contributes to the reduction of the carbon footprint during training.
  arXiv  Detail & Related papers  (2023-11-09T01:27:54Z)
• Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Model [77.19693792957614]
  We propose to make neural machine translation (NMT) models quality-aware by training them to estimate the quality of their own output. We obtain quality gains similar or even superior to quality reranking approaches, but with the efficiency of single pass decoding.
  arXiv  Detail & Related papers  (2023-10-10T15:33:51Z)
• Speculative Decoding with Big Little Decoder [108.95187338417541]
  Big Little Decoder (BiLD) is a framework that can improve inference efficiency and latency for a wide range of text generation applications. On an NVIDIA T4 GPU, our framework achieves a speedup of up to 2.12x speedup with minimal generation quality degradation. Our framework is fully plug-and-play and can be applied without any modifications in the training process or model architecture.
  arXiv  Detail & Related papers  (2023-02-15T18:55:29Z)
• ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference [70.36083572306839]
  This paper proposes a new training and inference paradigm for re-ranking. We finetune a pretrained encoder-decoder model using in the form of document to query generation. We show that this encoder-decoder architecture can be decomposed into a decoder-only language model during inference.
  arXiv  Detail & Related papers  (2022-04-25T06:26:29Z)
• E-LANG: Energy-Based Joint Inferencing of Super and Swift Language Models [9.36591003178585]
  This paper proposes an effective dynamic inference approach, called E-Lang, which distributes the inference between large accurate Super-models and light-weight Swift models. E-Lang is easily adoptable and architecture agnostic. Unlike existing methods that are only applicable to encoder-only backbones and classification tasks, our method also works for encoder-decoder structures and sequence-to-sequence tasks such as translation.
  arXiv  Detail & Related papers  (2022-03-01T21:21:27Z)
• Tiny Neural Models for Seq2Seq [0.0]
  We propose a projection based encoder-decoder model referred to as pQRNN-MAtt. The resulting quantized models are less than 3.5MB in size and are well suited for on-device latency critical applications. We show that on MTOP, a challenging multilingual semantic parsing dataset, the average model performance surpasses LSTM based seq2seq model that uses pre-trained embeddings despite being 85x smaller.
  arXiv  Detail & Related papers  (2021-08-07T00:39:42Z)
• Learning to Encode Position for Transformer with Continuous Dynamical Model [88.69870971415591]
  We introduce a new way of learning to encode position information for non-recurrent models, such as Transformer models. We model the evolution of encoded results along position index by such a dynamical system.
  arXiv  Detail & Related papers  (2020-03-13T00:41:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.