Decoupled Transformer for Scalable Inference in Open-domain Question Answering

• URL: http://arxiv.org/abs/2108.02765v1
• Date: Thu, 5 Aug 2021 17:53:40 GMT
• Title: Decoupled Transformer for Scalable Inference in Open-domain Question Answering
• Authors: Haytham ElFadeel and Stan Peshterliev
• Abstract summary: Large transformer models, such as BERT, achieve state-of-the-art results in machine reading comprehension (MRC) for open-domain question answering (QA) In experiments on the SQUAD 2.0 dataset, a decoupled transformer reduces the computational cost and latency of open-domain MRC by 30-40% with only 1.2 points worse F1-score compared to a standard transformer.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large transformer models, such as BERT, achieve state-of-the-art results in machine reading comprehension (MRC) for open-domain question answering (QA). However, transformers have a high computational cost for inference which makes them hard to apply to online QA systems for applications like voice assistants. To reduce computational cost and latency, we propose decoupling the transformer MRC model into input-component and cross-component. The decoupling allows for part of the representation computation to be performed offline and cached for online use. To retain the decoupled transformer accuracy, we devised a knowledge distillation objective from a standard transformer model. Moreover, we introduce learned representation compression layers which help reduce by four times the storage requirement for the cache. In experiments on the SQUAD 2.0 dataset, a decoupled transformer reduces the computational cost and latency of open-domain MRC by 30-40% with only 1.2 points worse F1-score compared to a standard transformer.

Related papers

• Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference [0.30104001512119216]
  Building models with fast and energy-efficient inference is imperative to enable a variety of transformer-based applications. We build on an approach for learning LUT networks directly via an Extended Finite Difference method. This allows for a computational and energy-efficient inference solution for transformer-based models.
  arXiv  Detail & Related papers  (2024-11-04T05:38:56Z)
• ENACT: Entropy-based Clustering of Attention Input for Improving the Computational Performance of Object Detection Transformers [0.0]
  Transformers demonstrate competitive performance in terms of precision on the problem of vision-based object detection. We propose to cluster the transformer input on the basis of its entropy. Clustering reduces the size of data given as input to the transformer and therefore reduces training time and GPU memory usage.
  arXiv  Detail & Related papers  (2024-09-11T18:03:59Z)
• Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching [56.286064975443026]
  We make an interesting and somehow surprising observation: the computation of a large proportion of layers in the diffusion transformer, through a caching mechanism, can be readily removed even without updating the model parameters. We introduce a novel scheme, named Learningto-Cache (L2C), that learns to conduct caching in a dynamic manner for diffusion transformers. Experimental results show that L2C largely outperforms samplers such as DDIM and DPM-r, alongside prior cache-based methods at the same inference speed.
  arXiv  Detail & Related papers  (2024-06-03T18:49:57Z)
• MoEUT: Mixture-of-Experts Universal Transformers [75.96744719516813]
  Universal Transformers (UTs) have advantages over standard Transformers in learning compositional generalizations. Layer-sharing drastically reduces the parameter count compared to the non-shared model with the same dimensionality. No previous work has succeeded in proposing a shared-layer Transformer design that is competitive in parameter count-dominated tasks such as language modeling.
  arXiv  Detail & Related papers  (2024-05-25T03:24:32Z)
• Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation [58.4650849317274]
  Volumetric Aggregation with Transformers (VAT) is a cost aggregation network for few-shot segmentation. VAT attains state-of-the-art performance for semantic correspondence as well, where cost aggregation also plays a central role.
  arXiv  Detail & Related papers  (2022-07-22T04:10:30Z)
• Resource-Efficient Separation Transformer [14.666016177212837]
  This paper explores Transformer-based speech separation with a reduced computational cost. Our main contribution is the development of the Resource-Efficient Separation Transformer (RE-SepFormer), a self-attention-based architecture. The RE-SepFormer reaches a competitive performance on the popular WSJ0-2Mix and WHAM! datasets in both causal and non-causal settings.
  arXiv  Detail & Related papers  (2022-06-19T23:37:24Z)
• Efficient Decoder-free Object Detection with Transformers [75.00499377197475]
  Vision transformers (ViTs) are changing the landscape of object detection approaches. We propose a decoder-free fully transformer-based (DFFT) object detector. DFFT_SMALL achieves high efficiency in both training and inference stages.
  arXiv  Detail & Related papers  (2022-06-14T13:22:19Z)
• Dual-Flattening Transformers through Decomposed Row and Column Queries for Semantic Segmentation [50.321277476317974]
  We propose a Dual-Flattening Transformer (DFlatFormer) to enable high-resolution output. Experiments on ADE20K and Cityscapes datasets demonstrate the superiority of the proposed dual-flattening transformer architecture.
  arXiv  Detail & Related papers  (2022-01-22T22:38:15Z)
• Quality and Cost Trade-offs in Passage Re-ranking Task [0.0]
  The paper is devoted to the problem of how to choose the right architecture for the ranking step of the information retrieval pipeline. We investigated several late-interaction models such as Colbert and Poly-encoder architectures along with their modifications. Also, we took care of the memory footprint of the search index and tried to apply the learning-to-hash method to binarize the output vectors from the transformer encoders.
  arXiv  Detail & Related papers  (2021-11-18T19:47:45Z)
• TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting [6.393659160890665]
  We propose the concept of tightly-coupled convolutional Transformer(TCCT) and three TCCT architectures. Our experiments on real-world datasets show that our TCCT architectures could greatly improve the performance of existing state-of-art Transformer models.
  arXiv  Detail & Related papers  (2021-08-29T08:49:31Z)
• The Cascade Transformer: an Application for Efficient Answer Sentence Selection [116.09532365093659]
  We introduce the Cascade Transformer, a technique to adapt transformer-based models into a cascade of rankers. When compared to a state-of-the-art transformer model, our approach reduces computation by 37% with almost no impact on accuracy.
  arXiv  Detail & Related papers  (2020-05-05T23:32:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.