Convergent Evolution in Neural Representation Space: Emergent Order in Deep Belief Networks

• URL: http://arxiv.org/abs/2608.05996v1
• Date: Thu, 06 Aug 2026 13:05:37 GMT
• Title: Convergent Evolution in Neural Representation Space: Emergent Order in Deep Belief Networks
• Abstract summary: Deep Belief Networks (DBNs) learn hierarchical generative models without class supervision.<n>We analyze successive layers of DBNs trained on MNIST, Fashion-MNIST, and KMNIST.<n>We find that class-specific clustering generally increases with depth across datasets and network widths.
• Score: 2.925652638976278
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Deep Belief Networks (DBNs) learn hierarchical generative models without class supervision. Here, we ask whether this purely unsupervised process nevertheless organizes internal representations according to the unknown data classes. We analyze successive layers of DBNs trained on MNIST, Fashion-MNIST, and KMNIST using the Generalized Discrimination Value (GDV), supervised probes applied only after training, a reconstruction-based measure of abstraction distance, effective dimensionality, and free sample generation. Remarkably, class-specific clustering generally increases with depth across datasets and network widths, although no label information is available during DBN training. Control experiments show that this effect depends on the learned feature structure and cannot be explained by random transformations, weight marginals, dimensionality reduction, or sigmoid saturation. The first hidden layers also frequently make class identity more accessible to linear and nonlinear probes. With greater depth, representations become increasingly compact and prototype-like as neurons acquire correlated feature directions. At the same time, GDV and probe accuracy reveal complementary aspects of class structure: improved average clustering can coexist with reduced accessibility for a few difficult class pairs. These findings demonstrate that layer-wise generative learning can spontaneously uncover and progressively amplify class-related structure in unlabeled data.

Related papers

• Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithm [4.838965998214037]
  Group-Contrastive Forward-Forward (GCFF) is a forward-forward training algorithm that combines class-specific routing with within-class contrastive objectives.<n>On CLIP representations, a single trained GCFF module recovers monosemantic neurons whose abstraction increases progressively with depth.
  arXiv  Detail & Related papers  (2026-07-13T09:54:35Z)
• An unsupervised tour through the hidden pathways of deep neural networks [6.063903439185316]
  This thesis focuses on characterizing semantic content of hidden representations with unsupervised learning tools.<n>In Chapter 3, we study the evolution of the probability density across the hidden layers in some state-of-the-art deep neural networks.<n>In Chapter 4, we study the problem of generalization in deep neural networks.
  arXiv  Detail & Related papers  (2025-10-24T15:50:31Z)
• Concept-Guided Interpretability via Neural Chunking [64.6429903327095]
  We show that neural networks exhibit patterns in their raw population activity that mirror regularities in the training data.<n>We propose three methods to extract recurring chunks on a neural population level.<n>Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data.
  arXiv  Detail & Related papers  (2025-05-16T13:49:43Z)
• Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $μ$P Parametrization [66.03821840425539]
  In this paper, we investigate the training dynamics of $L$-layer neural networks using the tensor gradient program (SGD) framework.<n>We show that SGD enables these networks to learn linearly independent features that substantially deviate from their initial values.<n>This rich feature space captures relevant data information and ensures that any convergent point of the training process is a global minimum.
  arXiv  Detail & Related papers  (2025-03-12T17:33:13Z)
• Average gradient outer product as a mechanism for deep neural collapse [26.939895223897572]
  Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the data representations in the final layers of Deep Neural Networks (DNNs)<n>In this work, we introduce a data-dependent setting where DNC forms due to feature learning through the average gradient outer product (AGOP)<n>We show that the right singular vectors and values of the weights can be responsible for the majority of within-class variability collapse for neural networks trained in the feature learning regime.
  arXiv  Detail & Related papers  (2024-02-21T11:40:27Z)
• Unsupervised Spatial-Temporal Feature Enrichment and Fidelity Preservation Network for Skeleton based Action Recognition [20.07820929037547]
  Unsupervised skeleton based action recognition has achieved remarkable progress recently. Existing unsupervised learning methods suffer from severe overfitting problem. This paper presents an Unsupervised spatial-temporal Feature Enrichment and Fidelity Preservation framework to generate rich distributed features.
  arXiv  Detail & Related papers  (2024-01-25T09:24:07Z)
• URLOST: Unsupervised Representation Learning without Stationarity or Topology [26.010647961403148]
  We introduce a novel framework that learns from high-dimensional data without prior knowledge of stationarity and topology.<n>Our model, abbreviated as URLOST, combines a learnable self-organizing layer, spectral clustering, and a masked autoencoder.<n>We evaluate its effectiveness on three diverse data modalities including simulated biological vision data, neural recordings from the primary visual cortex, and gene expressions.
  arXiv  Detail & Related papers  (2023-10-06T18:00:02Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay [67.50637511633212]
  A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day.
  arXiv  Detail & Related papers  (2021-12-09T07:11:14Z)
• Redundant representations help generalization in wide neural networks [71.38860635025907]
  We study the last hidden layer representations of various state-of-the-art convolutional neural networks. We find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise.
  arXiv  Detail & Related papers  (2021-06-07T10:18:54Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• High-order structure preserving graph neural network for few-shot learning [10.296473510866228]
  Few-shot learning can find the latent structure information between the prior knowledge and the queried data by the similarity metric of meta-learning. Most existing methods try to model the similarity relationship of the samples in the intra tasks, and generalize the model to identify the new categories. The proposed high-order structure preserving graph neural network(HOSP-GNN) can explore the rich structure of the samples to predict the label of the queried data on graph.
  arXiv  Detail & Related papers  (2020-05-29T06:38:51Z)
• Internal representation dynamics and geometry in recurrent neural networks [10.016265742591674]
  We show how a vanilla RNN implements a simple classification task by analysing the dynamics of the network. We find that early internal representations are evocative of the real labels of the data but this information is not directly accessible to the output layer.
  arXiv  Detail & Related papers  (2020-01-09T23:19:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.