Learning Deep Representation with Energy-Based Self-Expressiveness for Subspace Clustering

• URL: http://arxiv.org/abs/2110.15037v1
• Date: Thu, 28 Oct 2021 11:51:08 GMT
• Title: Learning Deep Representation with Energy-Based Self-Expressiveness for Subspace Clustering
• Authors: Yanming Li, Changsheng Li, Shiye Wang, Ye Yuan, Guoren Wang
• Abstract summary: We propose a new deep subspace clustering framework, motivated by the energy-based models. Considering the powerful representation ability of the recently popular self-supervised learning, we attempt to leverage self-supervised representation learning to learn the dictionary.
• Score: 24.311754971064303
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep subspace clustering has attracted increasing attention in recent years. Almost all the existing works are required to load the whole training data into one batch for learning the self-expressive coefficients in the framework of deep learning. Although these methods achieve promising results, such a learning fashion severely prevents from the usage of deeper neural network architectures (e.g., ResNet), leading to the limited representation abilities of the models. In this paper, we propose a new deep subspace clustering framework, motivated by the energy-based models. In contrast to previous approaches taking the weights of a fully connected layer as the self-expressive coefficients, we propose to learn an energy-based network to obtain the self-expressive coefficients by mini-batch training. By this means, it is no longer necessary to load all data into one batch for learning, and it thus becomes a reality that we can utilize deeper neural network models for subspace clustering. Considering the powerful representation ability of the recently popular self-supervised learning, we attempt to leverage self-supervised representation learning to learn the dictionary. Finally, we propose a joint framework to learn both the self-expressive coefficients and dictionary simultaneously, and train the model in an end-to-end manner. The experiments are performed on three publicly available datasets, and extensive experimental results demonstrate our method can significantly outperform the other related approaches. For instance, on the three datasets, our method can averagely achieve $13.8\%$, $15.4\%$, $20.8\%$ improvements in terms of Accuracy, NMI, and ARI over SENet which is proposed very recently and obtains the second best results in the experiments.

Related papers

• BEND: Bagging Deep Learning Training Based on Efficient Neural Network Diffusion [56.9358325168226]
  We propose a Bagging deep learning training algorithm based on Efficient Neural network Diffusion (BEND) Our approach is simple but effective, first using multiple trained model weights and biases as inputs to train autoencoder and latent diffusion model. Our proposed BEND algorithm can consistently outperform the mean and median accuracies of both the original trained model and the diffused model.
  arXiv  Detail & Related papers  (2024-03-23T08:40:38Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• The Trifecta: Three simple techniques for training deeper Forward-Forward networks [0.0]
  We propose a collection of three techniques that synergize exceptionally well and drastically improve the Forward-Forward algorithm on deeper networks. Our experiments demonstrate that our models are on par with similarly structured, backpropagation-based models in both training speed and test accuracy on simple datasets.
  arXiv  Detail & Related papers  (2023-11-29T22:44:32Z)
• Reinforcement Learning Based Multi-modal Feature Fusion Network for Novel Class Discovery [47.28191501836041]
  In this paper, we employ a Reinforcement Learning framework to simulate the cognitive processes of humans. We also deploy a Member-to-Leader Multi-Agent framework to extract and fuse features from multi-modal information. We demonstrate the performance of our approach in both the 3D and 2D domains by employing the OS-MN40, OS-MN40-Miss, and Cifar10 datasets.
  arXiv  Detail & Related papers  (2023-08-26T07:55:32Z)
• Boosting Low-Data Instance Segmentation by Unsupervised Pre-training with Saliency Prompt [103.58323875748427]
  This work offers a novel unsupervised pre-training solution for low-data regimes. Inspired by the recent success of the Prompting technique, we introduce a new pre-training method that boosts QEIS models. Experimental results show that our method significantly boosts several QEIS models on three datasets.
  arXiv  Detail & Related papers  (2023-02-02T15:49:03Z)
• Deep Dependency Networks for Multi-Label Classification [24.24496964886951]
  We show that the performance of previous approaches that combine Markov Random Fields with neural networks can be modestly improved. We propose a new modeling framework called deep dependency networks, which augments a dependency network. Despite its simplicity, jointly learning this new architecture yields significant improvements in performance.
  arXiv  Detail & Related papers  (2023-02-01T17:52:40Z)
• Deep networks for system identification: a Survey [56.34005280792013]
  System identification learns mathematical descriptions of dynamic systems from input-output data. Main aim of the identified model is to predict new data from previous observations. We discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks.
  arXiv  Detail & Related papers  (2023-01-30T12:38:31Z)
• A Deep Dive into Deep Cluster [0.2578242050187029]
  DeepCluster is a simple and scalable unsupervised pretraining of visual representations. We show that DeepCluster convergence and performance depend on the interplay between the quality of the randomly filters of the convolutional layer and the selected number of clusters.
  arXiv  Detail & Related papers  (2022-07-24T22:55:09Z)
• Learning a Self-Expressive Network for Subspace Clustering [15.096251922264281]
  We propose a novel framework for subspace clustering, termed Self-Expressive Network (SENet), which employs a properly designed neural network to learn a self-expressive representation of the data. Our SENet can not only learn the self-expressive coefficients with desired properties on the training data, but also handle out-of-sample data. In particular, SENet yields highly competitive performance on MNIST, Fashion MNIST and Extended MNIST and state-of-the-art performance on CIFAR-10.
  arXiv  Detail & Related papers  (2021-10-08T18:06:06Z)
• Self-supervised Knowledge Distillation for Few-shot Learning [123.10294801296926]
  Few shot learning is a promising learning paradigm due to its ability to learn out of order distributions quickly with only a few samples. We propose a simple approach to improve the representation capacity of deep neural networks for few-shot learning tasks. Our experiments show that, even in the first stage, self-supervision can outperform current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-17T11:27:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.