Improved Representation Learning Through Tensorized Autoencoders

• URL: http://arxiv.org/abs/2212.01046v1
• Date: Fri, 2 Dec 2022 09:29:48 GMT
• Title: Improved Representation Learning Through Tensorized Autoencoders
• Authors: Pascal Mattia Esser, Satyaki Mukherjee, Mahalakshmi Sabanayagam, Debarghya Ghoshdastidar
• Abstract summary: Autoencoders (AE) are widely used in practice for unsupervised representation learning. We propose a meta-algorithm that can be used to extend an arbitrary AE architecture to a tensorized version (TAE) We prove that TAE can recover the principle components of the different clusters in contrast to principle component of the entire data recovered by a standard AE.
• Score: 7.056005298953332
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The central question in representation learning is what constitutes a good or meaningful representation. In this work we argue that if we consider data with inherent cluster structures, where clusters can be characterized through different means and covariances, those data structures should be represented in the embedding as well. While Autoencoders (AE) are widely used in practice for unsupervised representation learning, they do not fulfil the above condition on the embedding as they obtain a single representation of the data. To overcome this we propose a meta-algorithm that can be used to extend an arbitrary AE architecture to a tensorized version (TAE) that allows for learning cluster-specific embeddings while simultaneously learning the cluster assignment. For the linear setting we prove that TAE can recover the principle components of the different clusters in contrast to principle component of the entire data recovered by a standard AE. We validated this on planted models and for general, non-linear and convolutional AEs we empirically illustrate that tensorizing the AE is beneficial in clustering and de-noising tasks.

Related papers

• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Class-Specific Semantic Reconstruction for Open Set Recognition [101.24781422480406]
  Open set recognition enables deep neural networks (DNNs) to identify samples of unknown classes. We propose a novel method, called Class-Specific Semantic Reconstruction (CSSR), that integrates the power of auto-encoder (AE) and prototype learning. Results of experiments conducted on multiple datasets show that the proposed method achieves outstanding performance in both close and open set recognition.
  arXiv  Detail & Related papers  (2022-07-05T16:25:34Z)
• Understanding Dynamics of Nonlinear Representation Learning and Its Application [12.697842097171119]
  We study the dynamics of implicit nonlinear representation learning. We show that the data-architecture alignment condition is sufficient for the global convergence. We derive a new training framework, which satisfies the data-architecture alignment condition without assuming it.
  arXiv  Detail & Related papers  (2021-06-28T16:31:30Z)
• Integrating Auxiliary Information in Self-supervised Learning [94.11964997622435]
  We first observe that the auxiliary information may bring us useful information about data structures. We present to construct data clusters according to the auxiliary information. We show that Cl-InfoNCE may be a better approach to leverage the data clustering information.
  arXiv  Detail & Related papers  (2021-06-05T11:01:15Z)
• You Never Cluster Alone [150.94921340034688]
  We extend the mainstream contrastive learning paradigm to a cluster-level scheme, where all the data subjected to the same cluster contribute to a unified representation. We define a set of categorical variables as clustering assignment confidence, which links the instance-level learning track with the cluster-level one. By reparametrizing the assignment variables, TCC is trained end-to-end, requiring no alternating steps.
  arXiv  Detail & Related papers  (2021-06-03T14:59:59Z)
• Towards Uncovering the Intrinsic Data Structures for Unsupervised Domain Adaptation using Structurally Regularized Deep Clustering [119.88565565454378]
  Unsupervised domain adaptation (UDA) is to learn classification models that make predictions for unlabeled data on a target domain. We propose a hybrid model of Structurally Regularized Deep Clustering, which integrates the regularized discriminative clustering of target data with a generative one. Our proposed H-SRDC outperforms all the existing methods under both the inductive and transductive settings.
  arXiv  Detail & Related papers  (2020-12-08T08:52:00Z)
• Deep Autoencoders: From Understanding to Generalization Guarantees [17.180863761413004]
  We take a step towards a better understanding of the underlying phenomena of Deep Autoencoders (AEs) In particular, we interpret how AEs approximate the data manifold by exploiting their continuous piecewise affine structure. We derive two new regularizations that enable AEs to capture the inherent symmetry in the data.
  arXiv  Detail & Related papers  (2020-09-20T21:01:18Z)
• Dual Adversarial Auto-Encoders for Clustering [152.84443014554745]
  We propose Dual Adversarial Auto-encoder (Dual-AAE) for unsupervised clustering. By performing variational inference on the objective function of Dual-AAE, we derive a new reconstruction loss which can be optimized by training a pair of Auto-encoders. Experiments on four benchmarks show that Dual-AAE achieves superior performance over state-of-the-art clustering methods.
  arXiv  Detail & Related papers  (2020-08-23T13:16:34Z)
• Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction [32.21975128854042]
  We propose the principle of Maximal Coding Rate Reduction ($textMCR2$), an information-theoretic measure that maximizes the coding rate difference between the whole dataset and the sum of each individual class. We clarify its relationships with most existing frameworks such as cross-entropy, information bottleneck, information gain, contractive and contrastive learning, and provide theoretical guarantees for learning diverse and discriminative features.
  arXiv  Detail & Related papers  (2020-06-15T17:23:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.