Efficient Sample-Specific Encoder Perturbations

• URL: http://arxiv.org/abs/2405.01601v1
• Date: Wed, 1 May 2024 08:55:16 GMT
• Title: Efficient Sample-Specific Encoder Perturbations
• Authors: Yassir Fathullah, Mark J. F. Gales,
• Abstract summary: We show that a small proxy network can be used to find a sample-by-sample perturbation of the encoder output of a frozen foundation model. Results show consistent improvements in performance evaluated through COMET and WER respectively.
• Score: 37.84914870036184
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Encoder-decoder foundation models have displayed state-of-the-art performance on a range of autoregressive sequence tasks. This paper proposes a simple and lightweight modification to such systems to control the behaviour according to a specific attribute of interest. This paper proposes a novel inference-efficient approach to modifying the behaviour of an encoder-decoder system according to a specific attribute of interest. Specifically, we show that a small proxy network can be used to find a sample-by-sample perturbation of the encoder output of a frozen foundation model to trigger the decoder to generate improved decodings. This work explores a specific realization of this framework focused on improving the COMET performance of Flan-T5 on Machine Translation and the WER of Whisper foundation models on Speech Recognition. Results display consistent improvements in performance evaluated through COMET and WER respectively. Furthermore, experiments also show that the proxies are robust to the exact nature of the data used to train them and can extend to other domains.

Related papers

• $ε$-VAE: Denoising as Visual Decoding [61.29255979767292]
  In generative modeling, tokenization simplifies complex data into compact, structured representations, creating a more efficient, learnable space. Current visual tokenization methods rely on a traditional autoencoder framework, where the encoder compresses data into latent representations, and the decoder reconstructs the original input. We propose denoising as decoding, shifting from single-step reconstruction to iterative refinement. Specifically, we replace the decoder with a diffusion process that iteratively refines noise to recover the original image, guided by the latents provided by the encoder. We evaluate our approach by assessing both reconstruction (rFID) and generation quality (
  arXiv  Detail & Related papers  (2024-10-05T08:27:53Z)
• PAGE: Parametric Generative Explainer for Graph Neural Network [16.350208494261913]
  PAGE is capable of providing faithful explanations for any graph neural network without necessitating prior knowledge or internal details. We introduce an additional discriminator to capture the causality between latent causal features and the model's output. Compared to existing methods, PAGE operates at the sample scale rather than nodes or edges.
  arXiv  Detail & Related papers  (2024-08-26T06:39:49Z)
• Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
  Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs) GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations. We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
  arXiv  Detail & Related papers  (2023-04-22T15:32:59Z)
• Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder [75.03283861464365]
  The seq2seq task aims at generating the target sequence based on the given input source sequence. Traditionally, most of the seq2seq task is resolved by an encoder to encode the source sequence and a decoder to generate the target text. Recently, a bunch of new approaches have emerged that apply decoder-only language models directly to the seq2seq task.
  arXiv  Detail & Related papers  (2023-04-08T15:44:29Z)
• String-based Molecule Generation via Multi-decoder VAE [56.465033997245776]
  We investigate the problem of string-based molecular generation via variational autoencoders (VAEs) We propose a simple, yet effective idea to improve the performance of VAE for the task. In our experiments, the proposed VAE model particularly performs well for generating a sample from out-of-domain distribution.
  arXiv  Detail & Related papers  (2022-08-23T03:56:30Z)
• Hyperdecoders: Instance-specific decoders for multi-task NLP [9.244884318445413]
  We investigate input-conditioned hypernetworks for multi-tasking in NLP. We generate parameter-efficient adaptations for a decoder using a hypernetwork conditioned on the output of an encoder.
  arXiv  Detail & Related papers  (2022-03-15T22:39:53Z)
• AttentionHTR: Handwritten Text Recognition Based on Attention Encoder-Decoder Networks [0.0]
  This work proposes an attention-based sequence-to-sequence model for handwritten word recognition. It exploits models pre-trained on scene text images as a starting point towards tailoring the handwriting recognition models. The effectiveness of the proposed end-to-end HTR system has been empirically evaluated on a novel multi-writer dataset.
  arXiv  Detail & Related papers  (2022-01-23T22:48:36Z)
• Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems [38.672160430296536]
  Transformer models have achieved state-of-the-art results in a wide range of NLP tasks including summarization. Previous work has focused on one important bottleneck, the quadratic self-attention mechanism in the encoder. This work focuses on the transformer's encoder-decoder attention mechanism.
  arXiv  Detail & Related papers  (2021-09-08T19:32:42Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.