Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia

• URL: http://arxiv.org/abs/2603.12316v1
• Date: Thu, 12 Mar 2026 18:00:01 GMT
• Title: Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia
• Abstract summary: We study fully-connected neural networks trained on MNIST, independently varying the dropout rate at both the training and evaluation stages.<n>We identify three distinct phases: eumentia (the network learns), dementia (the network has forgotten), and amentia (the network cannot learn)<n>Our results establish that dropout-induced pruning provides a concrete setting in which neural network behavior can be understood through the lens of statistical mechanics.
• Score: 0.18206461789819073
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Modern neural networks are heavily overparameterized, and pruning, which removes redundant neurons or connections, has emerged as a key approach to compressing them without sacrificing performance. However, while practical pruning methods are well developed, whether pruning induces sharp phase transitions in the neural networks and, if so, to what universality class they belong, remain open questions. To address this, we study fully-connected neural networks trained on MNIST, independently varying the dropout (i.e., removing neurons) rate at both the training and evaluation stages to map the phase diagram. We identify three distinct phases: eumentia (the network learns), dementia (the network has forgotten), and amentia (the network cannot learn), sharply distinguished by the power-law scaling of the cross-entropy loss with the training dataset size. {In the eumentia phase, the algebraic decay of the loss, as documented in the machine learning literature as neural scaling laws, is from the perspective of statistical mechanics the hallmark of quasi-long-range order.} We demonstrate that the transition between the eumentia and dementia phases is accompanied by scale invariance, with a diverging length scale that exhibits hallmarks of a Berezinskii-Kosterlitz-Thouless-like transition; the phase structure is robust across different network widths and depths. Our results establish that dropout-induced pruning provides a concrete setting in which neural network behavior can be understood through the lens of statistical mechanics.

Related papers

• An unsupervised tour through the hidden pathways of deep neural networks [6.063903439185316]
  This thesis focuses on characterizing semantic content of hidden representations with unsupervised learning tools.<n>In Chapter 3, we study the evolution of the probability density across the hidden layers in some state-of-the-art deep neural networks.<n>In Chapter 4, we study the problem of generalization in deep neural networks.
  arXiv  Detail & Related papers  (2025-10-24T15:50:31Z)
• The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions [51.68215326304272]
  We show that even small perturbations reliably cause otherwise identical training trajectories to diverge-an effect that diminishes rapidly over training time.<n>Our findings provide insights into neural network training stability, with practical implications for fine-tuning, model merging, and diversity of model ensembles.
  arXiv  Detail & Related papers  (2025-06-16T08:35:16Z)
• New Evidence of the Two-Phase Learning Dynamics of Neural Networks [59.55028392232715]
  We introduce an interval-wise perspective that compares network states across a time window.<n>We show that the response of the network to a perturbation exhibits a transition from chaotic to stable.<n>We also find that after this transition point the model's functional trajectory is confined to a narrow cone-shaped subset.
  arXiv  Detail & Related papers  (2025-05-20T04:03:52Z)
• Dissecting a Small Artificial Neural Network [0.0]
  We investigate the loss landscape and backpropagation dynamics of convergence for the simplest possible artificial neural network representing the logical exclusive-OR (XOR) gate.<n>Cross-sections of the loss landscape in the nine-dimensional parameter space are found to exhibit distinct features, which help understand why backpropagation achieves convergence toward zero loss.
  arXiv  Detail & Related papers  (2025-01-03T21:14:46Z)
• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• Addressing caveats of neural persistence with deep graph persistence [54.424983583720675]
  We find that the variance of network weights and spatial concentration of large weights are the main factors that impact neural persistence. We propose an extension of the filtration underlying neural persistence to the whole neural network instead of single layers. This yields our deep graph persistence measure, which implicitly incorporates persistent paths through the network and alleviates variance-related issues.
  arXiv  Detail & Related papers  (2023-07-20T13:34:11Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Neural network is heterogeneous: Phase matters more [10.812772606528172]
  In complex-valued neural networks, we show that among different types of pruning, the weight matrix with only phase information preserved achieves the best accuracy. The conclusion can be generalized to real-valued neural networks, where signs take the place of phases.
  arXiv  Detail & Related papers  (2021-11-03T04:30:20Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.