Progressive Learning and Disentanglement of Hierarchical Representations

• URL: http://arxiv.org/abs/2002.10549v1
• Date: Mon, 24 Feb 2020 21:19:38 GMT
• Title: Progressive Learning and Disentanglement of Hierarchical Representations
• Authors: Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali and Linwei Wang
• Abstract summary: We present a strategy to progressively learn independent hierarchical representations from high- to low-levels of abstractions. We quantitatively demonstrate the ability of the presented model to improve disentanglement in comparison to existing works.
• Score: 10.201945347770643
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning rich representation from data is an important task for deep generative models such as variational auto-encoder (VAE). However, by extracting high-level abstractions in the bottom-up inference process, the goal of preserving all factors of variations for top-down generation is compromised. Motivated by the concept of "starting small", we present a strategy to progressively learn independent hierarchical representations from high- to low-levels of abstractions. The model starts with learning the most abstract representation, and then progressively grow the network architecture to introduce new representations at different levels of abstraction. We quantitatively demonstrate the ability of the presented model to improve disentanglement in comparison to existing works on two benchmark data sets using three disentanglement metrics, including a new metric we proposed to complement the previously-presented metric of mutual information gap. We further present both qualitative and quantitative evidence on how the progression of learning improves disentangling of hierarchical representations. By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the first attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representations.

Related papers

• Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning [64.1745161657794]
  Domain-Incremental Learning (DIL) involves the progressive adaptation of a model to new concepts across different domains. Recent advances in pre-trained models provide a solid foundation for DIL. However, learning new concepts often results in the catastrophic forgetting of pre-trained knowledge. We propose DUal ConsolidaTion (Duct) to unify and consolidate historical knowledge.
  arXiv  Detail & Related papers  (2024-10-01T17:58:06Z)
• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• Self-supervised Learning of Dense Hierarchical Representations for Medical Image Segmentation [2.2265038612930663]
  This paper demonstrates a self-supervised framework for learning voxel-wise coarse-to-fine representations tailored for dense downstream tasks. We devise a training strategy that balances the contributions of features from multiple scales, ensuring that the learned representations capture both coarse and fine-grained details.
  arXiv  Detail & Related papers  (2024-01-12T09:47:17Z)
• Flow Factorized Representation Learning [109.51947536586677]
  We introduce a generative model which specifies a distinct set of latent probability paths that define different input transformations. We show that our model achieves higher likelihoods on standard representation learning benchmarks while simultaneously being closer to approximately equivariant models.
  arXiv  Detail & Related papers  (2023-09-22T20:15:37Z)
• Multi-View Class Incremental Learning [57.14644913531313]
  Multi-view learning (MVL) has gained great success in integrating information from multiple perspectives of a dataset to improve downstream task performance. This paper investigates a novel paradigm called multi-view class incremental learning (MVCIL), where a single model incrementally classifies new classes from a continual stream of views.
  arXiv  Detail & Related papers  (2023-06-16T08:13:41Z)
• Integrating Contrastive Learning with Dynamic Models for Reinforcement Learning from Images [31.413588478694496]
  We argue that explicitly improving Markovianity of the learned embedding is desirable. We propose a self-supervised representation learning method which integrates contrastive learning with dynamic models.
  arXiv  Detail & Related papers  (2022-03-02T14:39:17Z)
• Online Deep Learning based on Auto-Encoder [4.128388784932455]
  We propose a two-phase Online Deep Learning based on Auto-Encoder (ODLAE) Based on auto-encoder, considering reconstruction loss, we extract abstract hierarchical latent representations of instances. We devise two fusion strategies: the output-level fusion strategy, which is obtained by fusing the classification results of each hidden layer; and feature-level fusion strategy, which is leveraged self-attention mechanism to fusion every hidden layer output.
  arXiv  Detail & Related papers  (2022-01-19T02:14:57Z)
• DER: Dynamically Expandable Representation for Class Incremental Learning [30.573653645134524]
  We address the problem of class incremental learning, which is a core step towards achieving adaptive vision intelligence. We propose a novel two-stage learning approach that utilizes a dynamically expandable representation for more effective incremental concept modeling. We conduct extensive experiments on the three class incremental learning benchmarks and our method consistently outperforms other methods with a large margin.
  arXiv  Detail & Related papers  (2021-03-31T03:16:44Z)
• Two-Level Adversarial Visual-Semantic Coupling for Generalized Zero-shot Learning [21.89909688056478]
  We propose a new two-level joint idea to augment the generative network with an inference network during training. This provides strong cross-modal interaction for effective transfer of knowledge between visual and semantic domains. We evaluate our approach on four benchmark datasets against several state-of-the-art methods, and show its performance.
  arXiv  Detail & Related papers  (2020-07-15T15:34:09Z)
• Concept Learners for Few-Shot Learning [76.08585517480807]
  We propose COMET, a meta-learning method that improves generalization ability by learning to learn along human-interpretable concept dimensions. We evaluate our model on few-shot tasks from diverse domains, including fine-grained image classification, document categorization and cell type annotation.
  arXiv  Detail & Related papers  (2020-07-14T22:04:17Z)
• High-Fidelity Synthesis with Disentangled Representation [60.19657080953252]
  We propose an Information-Distillation Generative Adrial Network (ID-GAN) for disentanglement learning and high-fidelity synthesis. Our method learns disentangled representation using VAE-based models, and distills the learned representation with an additional nuisance variable to the separate GAN-based generator for high-fidelity synthesis. Despite the simplicity, we show that the proposed method is highly effective, achieving comparable image generation quality to the state-of-the-art methods using the disentangled representation.
  arXiv  Detail & Related papers  (2020-01-13T14:39:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.