Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction
- URL: http://arxiv.org/abs/2111.06636v1
- Date: Fri, 12 Nov 2021 10:06:08 GMT
- Title: Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction
- Authors: Xili Dai, Shengbang Tong, Mingyang Li, Ziyang Wu, Kwan Ho Ryan Chan,
Pengyuan Zhai, Yaodong Yu, Michael Psenka, Xiaojun Yuan, Heung Yeung Shum, Yi
Ma
- Abstract summary: This work proposes a new computational framework for learning an explicit generative model for real-world datasets.
In particular, we propose to learn em a closed-loop transcription between a multi-class multi-dimensional data distribution and a linear discriminative representation (LDR) in the feature space.
Our experiments on many benchmark imagery datasets demonstrate tremendous potential of this new closed-loop formulation.
- Score: 27.020835928724775
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This work proposes a new computational framework for learning an explicit
generative model for real-world datasets. In particular we propose to learn
{\em a closed-loop transcription} between a multi-class multi-dimensional data
distribution and a { linear discriminative representation (LDR)} in the feature
space that consists of multiple independent multi-dimensional linear subspaces.
In particular, we argue that the optimal encoding and decoding mappings sought
can be formulated as the equilibrium point of a {\em two-player minimax game
between the encoder and decoder}. A natural utility function for this game is
the so-called {\em rate reduction}, a simple information-theoretic measure for
distances between mixtures of subspace-like Gaussians in the feature space. Our
formulation draws inspiration from closed-loop error feedback from control
systems and avoids expensive evaluating and minimizing approximated distances
between arbitrary distributions in either the data space or the feature space.
To a large extent, this new formulation unifies the concepts and benefits of
Auto-Encoding and GAN and naturally extends them to the settings of learning a
{\em both discriminative and generative} representation for multi-class and
multi-dimensional real-world data. Our extensive experiments on many benchmark
imagery datasets demonstrate tremendous potential of this new closed-loop
formulation: under fair comparison, visual quality of the learned decoder and
classification performance of the encoder is competitive and often better than
existing methods based on GAN, VAE, or a combination of both. We notice that
the so learned features of different classes are explicitly mapped onto
approximately {\em independent principal subspaces} in the feature space; and
diverse visual attributes within each class are modeled by the {\em independent
principal components} within each subspace.
Related papers
- Unsupervised Manifold Linearizing and Clustering [19.879641608165887]
We propose to optimize the Maximal Coding Reduction metric with respect to both the data representation and a novel doubly cluster membership.
Experiments on CIFAR-10, -20, -100, and TinyImageNet-200 datasets show that the proposed method is much more accurate and scalable than state-of-the-art deep clustering methods.
arXiv Detail & Related papers (2023-01-04T20:08:23Z) - Dynamic Latent Separation for Deep Learning [67.62190501599176]
A core problem in machine learning is to learn expressive latent variables for model prediction on complex data.
Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
arXiv Detail & Related papers (2022-10-07T17:56:53Z) - Semi-Supervised Manifold Learning with Complexity Decoupled Chart
Autoencoders [65.2511270059236]
This work introduces a chart autoencoder with an asymmetric encoding-decoding process that can incorporate additional semi-supervised information such as class labels.
We discuss the theoretical approximation power of such networks that essentially depends on the intrinsic dimension of the data manifold.
arXiv Detail & Related papers (2022-08-22T19:58:03Z) - Multi-view hierarchical Variational AutoEncoders with Factor Analysis
latent space [67.60224656603823]
We propose a novel method to combine multiple Variational AutoEncoders with a Factor Analysis latent space.
We create an interpretable hierarchical dependency between private and shared information.
This way, the novel model is able to simultaneously: (i) learn from multiple heterogeneous views, (ii) obtain an interpretable hierarchical shared space, and (iii) perform transfer learning between generative models.
arXiv Detail & Related papers (2022-07-19T10:46:02Z) - Few-Shot Non-Parametric Learning with Deep Latent Variable Model [50.746273235463754]
We propose Non-Parametric learning by Compression with Latent Variables (NPC-LV)
NPC-LV is a learning framework for any dataset with abundant unlabeled data but very few labeled ones.
We show that NPC-LV outperforms supervised methods on all three datasets on image classification in low data regime.
arXiv Detail & Related papers (2022-06-23T09:35:03Z) - Generalized One-Class Learning Using Pairs of Complementary Classifiers [41.64645294104883]
One-class learning is the classic problem of fitting a model to the data for which annotations are available only for a single class.
In this paper, we explore novel objectives for one-class learning, which we collectively refer to as Generalized One-class Discriminative Subspaces (GODS)
arXiv Detail & Related papers (2021-06-24T18:52:05Z) - Learning optimally separated class-specific subspace representations
using convolutional autoencoder [0.0]
We propose a novel convolutional autoencoder based architecture to generate subspace specific feature representations.
To demonstrate the effectiveness of the proposed approach, several experiments have been carried out on state-of-the-art machine learning datasets.
arXiv Detail & Related papers (2021-05-19T00:45:34Z) - Input-Output Balanced Framework for Long-tailed LiDAR Semantic
Segmentation [12.639524717464509]
We propose an input-output balanced framework to handle the issue of long-tailed distribution.
For the input space, we synthesize these tailed instances from mesh models and well simulate the position and density distribution of LiDAR scan.
For the output space, a multi-head block is proposed to group different categories based on their shapes and instance amounts.
arXiv Detail & Related papers (2021-03-26T05:42:11Z) - Switch Spaces: Learning Product Spaces with Sparse Gating [48.591045282317424]
We propose Switch Spaces, a data-driven approach for learning representations in product space.
We introduce sparse gating mechanisms that learn to choose, combine and switch spaces.
Experiments on knowledge graph completion and item recommendations show that the proposed switch space achieves new state-of-the-art performances.
arXiv Detail & Related papers (2021-02-17T11:06:59Z) - Joint and Progressive Subspace Analysis (JPSA) with Spatial-Spectral
Manifold Alignment for Semi-Supervised Hyperspectral Dimensionality Reduction [48.73525876467408]
We propose a novel technique for hyperspectral subspace analysis.
The technique is called joint and progressive subspace analysis (JPSA)
Experiments are conducted to demonstrate the superiority and effectiveness of the proposed JPSA on two widely-used hyperspectral datasets.
arXiv Detail & Related papers (2020-09-21T16:29:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.