Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation
- URL: http://arxiv.org/abs/2110.07002v1
- Date: Wed, 13 Oct 2021 19:30:40 GMT
- Title: Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation
- Authors: Florian Mai and James Henderson
- Abstract summary: We extend Mai et al.'s proposed Emb2Emb method to learn mappings in the embedding space of an autoencoder.
We propose Bag-of-AEs Autoencoders (BoV-AEs), which encode the text into a variable-size bag of vectors that grows with the size of the text.
This allows to encode and reconstruct much longer texts than standard autoencoders.
- Score: 18.59238482225795
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Text autoencoders are often used for unsupervised conditional text generation
by applying mappings in the latent space to change attributes to the desired
values. Recently, Mai et al. (2020) proposed Emb2Emb, a method to learn these
mappings in the embedding space of an autoencoder. However, their method is
restricted to autoencoders with a single-vector embedding, which limits how
much information can be retained. We address this issue by extending their
method to Bag-of-Vectors Autoencoders (BoV-AEs), which encode the text into a
variable-size bag of vectors that grows with the size of the text, as in
attention-based models. This allows to encode and reconstruct much longer texts
than standard autoencoders. Analogous to conventional autoencoders, we propose
regularization techniques that facilitate learning meaningful operations in the
latent space. Finally, we adapt for a training scheme that learns to map an
input bag to an output bag, including a novel loss function and neural
architecture. Our experimental evaluations on unsupervised sentiment transfer
and sentence summarization show that our method performs substantially better
than a standard autoencoder.
Related papers
- Triple-Encoders: Representations That Fire Together, Wire Together [51.15206713482718]
Contrastive Learning is a representation learning method that encodes relative distances between utterances into the embedding space via a bi-encoder.
This study introduces triple-encoders, which efficiently compute distributed utterance mixtures from these independently encoded utterances.
We find that triple-encoders lead to a substantial improvement over bi-encoders, and even to better zero-shot generalization than single-vector representation models.
arXiv Detail & Related papers (2024-02-19T18:06:02Z) - Regress Before Construct: Regress Autoencoder for Point Cloud
Self-supervised Learning [18.10704604275133]
Masked Autoencoders (MAE) have demonstrated promising performance in self-supervised learning for 2D and 3D computer vision.
We propose Point Regress AutoEncoder (Point-RAE), a new scheme for regressive autoencoders for point cloud self-supervised learning.
Our approach is efficient during pre-training and generalizes well on various downstream tasks.
arXiv Detail & Related papers (2023-09-25T17:23:33Z) - ConTextual Mask Auto-Encoder for Dense Passage Retrieval [49.49460769701308]
CoT-MAE is a simple yet effective generative pre-training method for dense passage retrieval.
It learns to compress the sentence semantics into a dense vector through self-supervised and context-supervised masked auto-encoding.
We conduct experiments on large-scale passage retrieval benchmarks and show considerable improvements over strong baselines.
arXiv Detail & Related papers (2022-08-16T11:17:22Z) - An Introduction to Autoencoders [0.0]
This article covers the mathematics and the fundamental concepts of autoencoders.
We will start with a general introduction to autoencoders, and we will discuss the role of the activation function in the output layer and the loss function.
arXiv Detail & Related papers (2022-01-11T11:55:32Z) - Sentence Bottleneck Autoencoders from Transformer Language Models [53.350633961266375]
We build a sentence-level autoencoder from a pretrained, frozen transformer language model.
We adapt the masked language modeling objective as a generative, denoising one, while only training a sentence bottleneck and a single-layer modified transformer decoder.
We demonstrate that the sentence representations discovered by our model achieve better quality than previous methods that extract representations from pretrained transformers on text similarity tasks, style transfer, and single-sentence classification tasks in the GLUE benchmark, while using fewer parameters than large pretrained models.
arXiv Detail & Related papers (2021-08-31T19:39:55Z) - Text Compression-aided Transformer Encoding [77.16960983003271]
We propose explicit and implicit text compression approaches to enhance the Transformer encoding.
backbone information, meaning the gist of the input text, is not specifically focused on.
Our evaluation on benchmark datasets shows that the proposed explicit and implicit text compression approaches improve results in comparison to strong baselines.
arXiv Detail & Related papers (2021-02-11T11:28:39Z) - Cross-Thought for Sentence Encoder Pre-training [89.32270059777025]
Cross-Thought is a novel approach to pre-training sequence encoder.
We train a Transformer-based sequence encoder over a large set of short sequences.
Experiments on question answering and textual entailment tasks demonstrate that our pre-trained encoder can outperform state-of-the-art encoders.
arXiv Detail & Related papers (2020-10-07T21:02:41Z) - Plug and Play Autoencoders for Conditional Text Generation [0.0]
We propose a method where any pretrained autoencoder can be used to train embedding-to-embedding.
This reduces the need for labeled training data for the task and makes the training procedure more efficient.
We show that our method performs better than or comparable to strong baselines while being up to four times faster.
arXiv Detail & Related papers (2020-10-06T19:18:06Z) - Double Backpropagation for Training Autoencoders against Adversarial
Attack [15.264115499966413]
This paper focuses on the adversarial attack on autoencoders.
We propose to adopt double backpropagation (DBP) to secure autoencoder such as VAE and DRAW.
arXiv Detail & Related papers (2020-03-04T05:12:27Z) - Learning Autoencoders with Relational Regularization [89.53065887608088]
A new framework is proposed for learning autoencoders of data distributions.
We minimize the discrepancy between the model and target distributions, with a emphrelational regularization
We implement the framework with two scalable algorithms, making it applicable for both probabilistic and deterministic autoencoders.
arXiv Detail & Related papers (2020-02-07T17:27:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.