Plug and Play Autoencoders for Conditional Text Generation

• URL: http://arxiv.org/abs/2010.02983v2
• Date: Mon, 12 Oct 2020 08:20:59 GMT
• Title: Plug and Play Autoencoders for Conditional Text Generation
• Authors: Florian Mai (1 and 2), Nikolaos Pappas (3), Ivan Montero (3), Noah A. Smith (3 and 4), James Henderson (1) ((1) Idiap Research Institute, (2) EPFL, (3) University of Washington, (4) Allen Institute for Artificial Intelligence)
• Abstract summary: We propose a method where any pretrained autoencoder can be used to train embedding-to-embedding. This reduces the need for labeled training data for the task and makes the training procedure more efficient. We show that our method performs better than or comparable to strong baselines while being up to four times faster.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text autoencoders are commonly used for conditional generation tasks such as style transfer. We propose methods which are plug and play, where any pretrained autoencoder can be used, and only require learning a mapping within the autoencoder's embedding space, training embedding-to-embedding (Emb2Emb). This reduces the need for labeled training data for the task and makes the training procedure more efficient. Crucial to the success of this method is a loss term for keeping the mapped embedding on the manifold of the autoencoder and a mapping which is trained to navigate the manifold by learning offset vectors. Evaluations on style transfer tasks both with and without sequence-to-sequence supervision show that our method performs better than or comparable to strong baselines while being up to four times faster.

Related papers

• Efficient Pre-training for Localized Instruction Generation of Videos [32.13509517228516]
  Procedural videos are instrumental in conveying step-by-step instructions. Process Transformer (ProcX) is a model for end-to-end step localization and instruction generation for procedural videos.
  arXiv  Detail & Related papers  (2023-11-27T16:07:37Z)
• How Much Training Data is Memorized in Overparameterized Autoencoders? An Inverse Problem Perspective on Memorization Evaluation [1.573034584191491]
  We propose an inverse problem perspective for the study of memorization. We use the trained autoencoder to implicitly define a regularizer for the particular training dataset that we aim to retrieve from. We show that our method significantly outperforms previous memorization-evaluation methods that recover training data from autoencoders.
  arXiv  Detail & Related papers  (2023-10-04T15:36:33Z)
• TransCoder: Towards Unified Transferable Code Representation Learning Inspired by Human Skills [31.75121546422898]
  We present TransCoder, a unified Transferable fine-tuning strategy for Code representation learning. We employ a tunable prefix encoder as the meta-learner to capture cross-task and cross-language transferable knowledge. Our method can lead to superior performance on various code-related tasks and encourage mutual reinforcement.
  arXiv  Detail & Related papers  (2023-05-23T06:59:22Z)
• MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrievers [140.0479479231558]
  In this work, we aim to unify a variety of pre-training tasks into a multi-task pre-trained model, namely MASTER. MASTER utilizes a shared-encoder multi-decoder architecture that can construct a representation bottleneck to compress the abundant semantic information across tasks into dense vectors.
  arXiv  Detail & Related papers  (2022-12-15T13:57:07Z)
• Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processing [66.92823764664206]
  We take a sober look into pre-trained speech encoders and rewire their representation space without requiring task-specific labels. Our experiments on 6 speech processing tasks, exhibit a significant convergence speedup during task fine-tuning as well as consistent task improvement.
  arXiv  Detail & Related papers  (2022-10-24T08:27:09Z)
• KRNet: Towards Efficient Knowledge Replay [50.315451023983805]
  A knowledge replay technique has been widely used in many tasks such as continual learning and continuous domain adaptation. We propose a novel and efficient knowledge recording network (KRNet) which directly maps an arbitrary sample identity number to the corresponding datum. Our KRNet requires significantly less storage cost for the latent codes and can be trained without the encoder sub-network.
  arXiv  Detail & Related papers  (2022-05-23T08:34:17Z)
• UniXcoder: Unified Cross-Modal Pre-training for Code Representation [65.6846553962117]
  We present UniXcoder, a unified cross-modal pre-trained model for programming language. We propose a one-to-one mapping method to transform AST in a sequence structure that retains all structural information from the tree. We evaluate UniXcoder on five code-related tasks over nine datasets.
  arXiv  Detail & Related papers  (2022-03-08T04:48:07Z)
• Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation [18.59238482225795]
  We extend Mai et al.'s proposed Emb2Emb method to learn mappings in the embedding space of an autoencoder. We propose Bag-of-AEs Autoencoders (BoV-AEs), which encode the text into a variable-size bag of vectors that grows with the size of the text. This allows to encode and reconstruct much longer texts than standard autoencoders.
  arXiv  Detail & Related papers  (2021-10-13T19:30:40Z)
• InferCode: Self-Supervised Learning of Code Representations by Predicting Subtrees [17.461451218469062]
  This paper proposes InferCode to overcome the limitation by adapting the self-language learning mechanism to build source code model. Subtrees in ASTs are treated with InferCode as the labels for training code representations without any human labeling effort or the overhead of expensive graph construction. Compared to previous code learning techniques applied to the same downstream tasks, such as Code2Vec, Code2Seq, ASTNN, higher performance results are achieved using our pre-trained InferCode model.
  arXiv  Detail & Related papers  (2020-12-13T10:33:41Z)
• Conditioned Text Generation with Transfer for Closed-Domain Dialogue Systems [65.48663492703557]
  We show how to optimally train and control the generation of intent-specific sentences using a conditional variational autoencoder. We introduce a new protocol called query transfer that allows to leverage a large unlabelled dataset.
  arXiv  Detail & Related papers  (2020-11-03T14:06:10Z)
• Cross-Thought for Sentence Encoder Pre-training [89.32270059777025]
  Cross-Thought is a novel approach to pre-training sequence encoder. We train a Transformer-based sequence encoder over a large set of short sequences. Experiments on question answering and textual entailment tasks demonstrate that our pre-trained encoder can outperform state-of-the-art encoders.
  arXiv  Detail & Related papers  (2020-10-07T21:02:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.