Uni-Encoder: A Fast and Accurate Response Selection Paradigm for Generation-Based Dialogue Systems

• URL: http://arxiv.org/abs/2106.01263v5
• Date: Mon, 15 May 2023 16:53:58 GMT
• Title: Uni-Encoder: A Fast and Accurate Response Selection Paradigm for Generation-Based Dialogue Systems
• Authors: Chiyu Song, Hongliang He, Haofei Yu, Pengfei Fang, Leyang Cui and Zhenzhong Lan
• Abstract summary: We develop a new encoding paradigm called Uni-Encoder. It keeps the full attention over each pair as in Cross-Encoder while only encoding the context once. It achieves new state-of-the-art results on four benchmark datasets with high computational efficiency.
• Score: 17.041715422600504
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sample-and-rank is a key decoding strategy for modern generation-based dialogue systems. It helps achieve diverse and high-quality responses by selecting an answer from a small pool of generated candidates. The current state-of-the-art ranking methods mainly use an encoding paradigm called Cross-Encoder, which separately encodes each context-candidate pair and ranks the candidates according to their fitness scores. However, Cross-Encoder repeatedly encodes the same lengthy context for each candidate, resulting in high computational costs. Poly-Encoder addresses the above problems by reducing the interaction between context and candidates, but with a price of performance drop. In this work, we develop a new paradigm called Uni-Encoder, that keeps the full attention over each pair as in Cross-Encoder while only encoding the context once, as in Poly-Encoder. Uni-Encoder encodes all the candidates with the context in one forward pass. We use the same positional embedding for all candidates to ensure they are treated equally and design a new attention mechanism to avoid confusion. Our Uni-Encoder can simulate other ranking paradigms using different attention and response concatenation methods. Extensive experiments show that our proposed paradigm achieves new state-of-the-art results on four benchmark datasets with high computational efficiency. For instance, it improves R10@1 by 2.9% with an approximately 4X faster inference speed on the Ubuntu V2 dataset.

Related papers

• Hold Me Tight: Stable Encoder-Decoder Design for Speech Enhancement [1.4037575966075835]
  1-D filters on raw audio are hard to train and often suffer from instabilities. We address these problems with hybrid solutions, combining theory-driven and data-driven approaches.
  arXiv  Detail & Related papers  (2024-08-30T15:49:31Z)
• Triple-Encoders: Representations That Fire Together, Wire Together [51.15206713482718]
  Contrastive Learning is a representation learning method that encodes relative distances between utterances into the embedding space via a bi-encoder. This study introduces triple-encoders, which efficiently compute distributed utterance mixtures from these independently encoded utterances. We find that triple-encoders lead to a substantial improvement over bi-encoders, and even to better zero-shot generalization than single-vector representation models.
  arXiv  Detail & Related papers  (2024-02-19T18:06:02Z)
• Question Answering Infused Pre-training of General-Purpose Contextualized Representations [70.62967781515127]
  We propose a pre-training objective based on question answering (QA) for learning general-purpose contextual representations. We accomplish this goal by training a bi-encoder QA model, which independently encodes passages and questions, to match the predictions of a more accurate cross-encoder model. We show large improvements over both RoBERTa-large and previous state-of-the-art results on zero-shot and few-shot paraphrase detection.
  arXiv  Detail & Related papers  (2021-06-15T14:45:15Z)
• Fast Interleaved Bidirectional Sequence Generation [90.58793284654692]
  We introduce a decoder that generates target words from the left-to-right and right-to-left directions simultaneously. We show that we can easily convert a standard architecture for unidirectional decoding into a bidirectional decoder. Our interleaved bidirectional decoder (IBDecoder) retains the model simplicity and training efficiency of the standard Transformer.
  arXiv  Detail & Related papers  (2020-10-27T17:38:51Z)
• Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder [64.55176104620848]
  We propose a novel NAR E2E-ST framework, Orthros, in which both NAR and autoregressive (AR) decoders are jointly trained on the shared speech encoder. The latter is used for selecting better translation among various length candidates generated from the former, which dramatically improves the effectiveness of a large length beam with negligible overhead. Experiments on four benchmarks show the effectiveness of the proposed method in improving inference speed while maintaining competitive translation quality.
  arXiv  Detail & Related papers  (2020-10-25T06:35:30Z)
• Dual Encoding for Video Retrieval by Text [49.34356217787656]
  We propose a dual deep encoding network that encodes videos and queries into powerful dense representations of their own. Our novelty is two-fold. First, different from prior art that resorts to a specific single-level encoder, the proposed network performs multi-level encoding. Second, different from a conventional common space learning algorithm which is either concept based or latent space based, we introduce hybrid space learning.
  arXiv  Detail & Related papers  (2020-09-10T15:49:39Z)
• Distilling Knowledge for Fast Retrieval-based Chat-bots [6.284464997330884]
  We propose a new cross-encoder architecture and transfer knowledge from this model to a bi-encoder model using distillation. This effectively boosts bi-encoder performance at no cost during inference time.
  arXiv  Detail & Related papers  (2020-04-23T09:41:37Z)
• Consistent Multiple Sequence Decoding [36.46573114422263]
  We introduce a consistent multiple sequence decoding architecture. This architecture allows for consistent and simultaneous decoding of an arbitrary number of sequences. We show the efficacy of our consistent multiple sequence decoder on the task of dense relational image captioning.
  arXiv  Detail & Related papers  (2020-04-02T00:43:54Z)
• Learning Autoencoders with Relational Regularization [89.53065887608088]
  A new framework is proposed for learning autoencoders of data distributions. We minimize the discrepancy between the model and target distributions, with a emphrelational regularization We implement the framework with two scalable algorithms, making it applicable for both probabilistic and deterministic autoencoders.
  arXiv  Detail & Related papers  (2020-02-07T17:27:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.