An Introduction to Autoencoders
- URL: http://arxiv.org/abs/2201.03898v1
- Date: Tue, 11 Jan 2022 11:55:32 GMT
- Title: An Introduction to Autoencoders
- Authors: Umberto Michelucci
- Abstract summary: This article covers the mathematics and the fundamental concepts of autoencoders.
We will start with a general introduction to autoencoders, and we will discuss the role of the activation function in the output layer and the loss function.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this article, we will look at autoencoders. This article covers the
mathematics and the fundamental concepts of autoencoders. We will discuss what
they are, what the limitations are, the typical use cases, and we will look at
some examples. We will start with a general introduction to autoencoders, and
we will discuss the role of the activation function in the output layer and the
loss function. We will then discuss what the reconstruction error is. Finally,
we will look at typical applications as dimensionality reduction,
classification, denoising, and anomaly detection. This paper contains the notes
of a PhD-level lecture on autoencoders given in 2021.
Related papers
- Triple-Encoders: Representations That Fire Together, Wire Together [51.15206713482718]
Contrastive Learning is a representation learning method that encodes relative distances between utterances into the embedding space via a bi-encoder.
This study introduces triple-encoders, which efficiently compute distributed utterance mixtures from these independently encoded utterances.
We find that triple-encoders lead to a substantial improvement over bi-encoders, and even to better zero-shot generalization than single-vector representation models.
arXiv Detail & Related papers (2024-02-19T18:06:02Z) - Rethinking Patch Dependence for Masked Autoencoders [92.37365660775171]
We re-examine inter-patch dependencies in the decoding mechanism of masked autoencoders (MAE)
We propose a novel pretraining framework: Cross-Attention Masked Autoencoders (CrossMAE)
arXiv Detail & Related papers (2024-01-25T18:49:57Z) - Decoder-Only or Encoder-Decoder? Interpreting Language Model as a
Regularized Encoder-Decoder [75.03283861464365]
The seq2seq task aims at generating the target sequence based on the given input source sequence.
Traditionally, most of the seq2seq task is resolved by an encoder to encode the source sequence and a decoder to generate the target text.
Recently, a bunch of new approaches have emerged that apply decoder-only language models directly to the seq2seq task.
arXiv Detail & Related papers (2023-04-08T15:44:29Z) - Deep Nonparametric Estimation of Intrinsic Data Structures by Chart
Autoencoders: Generalization Error and Robustness [11.441464617936173]
We employ chart autoencoders to encode data into low-dimensional latent features on a collection of charts.
By training autoencoders, we show that chart autoencoders can effectively denoise the input data with normal noise.
As a special case, our theory also applies to classical autoencoders, as long as the data manifold has a global parametrization.
arXiv Detail & Related papers (2023-03-17T10:01:32Z) - On a Mechanism Framework of Autoencoders [0.0]
This paper proposes a theoretical framework on the mechanism of autoencoders.
Results of ReLU autoencoders are generalized to some non-ReLU cases.
Compared to PCA and decision trees, the advantages of (generalized) autoencoders on dimensionality reduction and classification are demonstrated.
arXiv Detail & Related papers (2022-08-15T03:51:40Z) - Masked Autoencoders that Listen [79.99280830830854]
This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms.
Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers.
The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram.
arXiv Detail & Related papers (2022-07-13T17:59:55Z) - Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation [18.59238482225795]
We extend Mai et al.'s proposed Emb2Emb method to learn mappings in the embedding space of an autoencoder.
We propose Bag-of-AEs Autoencoders (BoV-AEs), which encode the text into a variable-size bag of vectors that grows with the size of the text.
This allows to encode and reconstruct much longer texts than standard autoencoders.
arXiv Detail & Related papers (2021-10-13T19:30:40Z) - Revisiting Role of Autoencoders in Adversarial Settings [32.22707594954084]
This paper presents the inherent property of adversarial robustness in the autoencoders.
We believe that our discovery of the adversarial robustness of the autoencoders can provide clues to the future research and applications for adversarial defense.
arXiv Detail & Related papers (2020-05-21T16:01:23Z) - On Sparsifying Encoder Outputs in Sequence-to-Sequence Models [90.58793284654692]
We take Transformer as the testbed and introduce a layer of gates in-between the encoder and the decoder.
The gates are regularized using the expected value of the sparsity-inducing L0penalty.
We investigate the effects of this sparsification on two machine translation and two summarization tasks.
arXiv Detail & Related papers (2020-04-24T16:57:52Z) - Autoencoders [43.991924654575975]
An autoencoder is a specific type of a neural network, which is mainly designed to encode the input into a compressed and meaningful representation, and then decode it back such that the reconstructed input is similar as possible to the original one.
This chapter surveys the different types of autoencoders that are mainly used today.
arXiv Detail & Related papers (2020-03-12T19:38:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.