Spectral structural distortion reveals redundant neurons in neural networks

• URL: http://arxiv.org/abs/2605.18860v2
• Date: Wed, 20 May 2026 06:09:53 GMT
• Title: Spectral structural distortion reveals redundant neurons in neural networks
• Abstract summary: We show that neuronal redundancy can be characterized by weak participation in the spectral structural distortion induced by layer-wise representation transformations.<n>For each hidden layer of a trained network, we record pre-activation and post-activation hidden states, model neurons as graph nodes, and construct input-side and output-side graphs.<n>We then define a spectral structural importance score that measures the contribution of each neuron to the dominant graph-spectral distortion between these two relational structures.
• Score: 1.066048003460524
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Overparameterized neural networks often contain many removable neurons, yet what makes a neuron redundant remains poorly understood. Existing pruning criteria commonly rely on local quantities such as weight magnitude, activation strength, or gradient sensitivity, but these measures provide limited insight into the structural role of a neuron in the transformation performed by a layer. Here we show that neuronal redundancy can be characterized by weak participation in the spectral structural distortion induced by layer-wise representation transformations. For each hidden layer of a trained network, we record pre-activation and post-activation hidden states, model neurons as graph nodes, and construct input-side and output-side graphs that describe neuron-level relational structure before and after the layer transformation. We then define a spectral structural importance score that measures the contribution of each neuron to the dominant graph-spectral distortion between these two relational structures. Low-participation neurons are treated as structurally redundant and removed through an iterative pruning process in which scores are recomputed after each structural change. No parameter updates are performed during intermediate pruning rounds; after the target parameter reduction is reached, a single recovery fine-tuning stage is applied to the compact model. Direct ablation analysis and experiments across conventional neural networks, encoder-only Transformers, and decoder-only language models show that this graph-spectral criterion identifies removable neurons and Transformer units while preserving task performance after compression. These results suggest that neural redundancy is not merely a consequence of small weights or weak activations, but can be understood through weak participation in the spectral distortion of layer-wise relational structure.

Related papers

• Sparse Inter-Layer Dependencies of Transformer FFN Neurons [32.00544060412007]
  We study whether the activation of an FFN neuron can be explained by a sparse set of preceding neuron activations and attention outputs.<n>We introduce a training-free attribution method that estimates the relative influence of upstream neurons and attention outputs on a target neuron's activation.
  arXiv  Detail & Related papers  (2026-07-13T15:04:50Z)
• On De-Individuated Neurons: Continuous Symmetries Enable Dynamic Topologies [0.0]
  This paper introduces a novel methodology for computation by leveraging a new symmetry-principled class of primitives, isotropic activation functions.<n>It enables real-time neuronal growth and shrinkage of the architectures in response to task demand.
  arXiv  Detail & Related papers  (2026-02-26T12:47:14Z)
• Neuronal Group Communication for Efficient Neural representation [85.36421257648294]
  This paper addresses the question of how to build large neural systems that learn efficient, modular, and interpretable representations.<n>We propose Neuronal Group Communication (NGC), a theory-driven framework that reimagines a neural network as a dynamical system of interacting neuronal groups.<n>NGC treats weights as transient interactions between embedding-like neuronal states, with neural computation unfolding through iterative communication among groups of neurons.
  arXiv  Detail & Related papers  (2025-10-19T14:23:35Z)
• Pixel Embedding Method for Tubular Neurite Segmentation [14.942761606240683]
  We introduce a deep network that outputs pixel level embedding vectors and design a corresponding loss function.<n>We also develop an end to end pipeline that directly maps raw neuronal images to SWC formatted neuron structure trees.<n> Experiments on our fMOST imaging dataset demonstrate that, compared to several classical methods, our approach significantly reduces the error rate in neuronal topology reconstruction.
  arXiv  Detail & Related papers  (2025-07-31T09:11:15Z)
• Confidence Regulation Neurons in Language Models [91.90337752432075]
  This study investigates the mechanisms by which large language models represent and regulate uncertainty in next-token predictions. Entropy neurons are characterized by an unusually high weight norm and influence the final layer normalization (LayerNorm) scale to effectively scale down the logits. token frequency neurons, which we describe here for the first time, boost or suppress each token's logit proportionally to its log frequency, thereby shifting the output distribution towards or away from the unigram distribution.
  arXiv  Detail & Related papers  (2024-06-24T01:31:03Z)
• Addressing caveats of neural persistence with deep graph persistence [54.424983583720675]
  We find that the variance of network weights and spatial concentration of large weights are the main factors that impact neural persistence. We propose an extension of the filtration underlying neural persistence to the whole neural network instead of single layers. This yields our deep graph persistence measure, which implicitly incorporates persistent paths through the network and alleviates variance-related issues.
  arXiv  Detail & Related papers  (2023-07-20T13:34:11Z)
• Quantifying the Optimization and Generalization Advantages of Graph Neural Networks Over Multilayer Perceptrons [50.33260238739837]
  Graph networks (GNNs) have demonstrated remarkable capabilities in learning from graph-structured data.<n>There remains a lack of analysis comparing GNNs and generalizations from an optimization and generalization perspective.
  arXiv  Detail & Related papers  (2023-06-24T10:21:11Z)
• Graph Neural Operators for Classification of Spatial Transcriptomics Data [1.408706290287121]
  We propose a study incorporating various graph neural network approaches to validate the efficacy of applying neural operators towards prediction of brain regions in mouse brain tissue samples. We were able to achieve an F1 score of nearly 72% for the graph neural operator approach which outperformed all baseline and other graph network approaches.
  arXiv  Detail & Related papers  (2023-02-01T18:32:06Z)
• PointNeuron: 3D Neuron Reconstruction via Geometry and Topology Learning of Point Clouds [18.738943602529805]
  We propose a novel framework for 3D neuron reconstruction. Our key idea is to use the geometric representation power of the point cloud to better explore the intrinsic structural information of neurons.
  arXiv  Detail & Related papers  (2022-10-15T14:11:56Z)
• A Step Towards Uncovering The Structure of Multistable Neural Networks [1.14219428942199]
  We study the structure of multistable recurrent neural networks. The activation function is simplified by a nonsmooth Heaviside step function. We derive how multistability is encoded within the network architecture.
  arXiv  Detail & Related papers  (2022-10-06T22:54:17Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• NeuRegenerate: A Framework for Visualizing Neurodegeneration [10.27276267081559]
  We introduce NeuRegenerate, a novel end-to-end framework for the prediction and visualization of changes in neural fiber morphology within a subject. To predict projections, we present neuReGANerator, a deep-learning network based on cycle-consistent generative adversarial network (cycleGAN) We show that neuReGANerator has a reconstruction accuracy of 94% in predicting neuronal structures.
  arXiv  Detail & Related papers  (2022-02-02T16:21:14Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Network [39.341620528427306]
  We develop a local convergence theory for mildly over- parameterized neural networks. We show that as long as the loss is already lower than a threshold, all student neurons converge to one of teacher neurons. Our result holds for any number of student neurons as long as it is at least as large as the number of teacher neurons.
  arXiv  Detail & Related papers  (2021-02-04T04:41:04Z)
• Single Neuron Segmentation using Graph-based Global Reasoning with Auxiliary Skeleton Loss from 3D Optical Microscope Images [30.539098538610013]
  We present an end-to-end segmentation network by jointly considering the local appearance and the global geometry traits. The evaluation results on the Janelia dataset from the BigNeuron project demonstrate that our proposed method exceeds the counterpart algorithms in performance.
  arXiv  Detail & Related papers  (2021-01-22T01:27:14Z)
• Topological obstructions in neural networks learning [67.8848058842671]
  We study global properties of the loss gradient function flow. We use topological data analysis of the loss function and its Morse complex to relate local behavior along gradient trajectories with global properties of the loss surface.
  arXiv  Detail & Related papers  (2020-12-31T18:53:25Z)
• Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
  Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
  arXiv  Detail & Related papers  (2020-11-18T18:52:08Z)
• Graph Structure of Neural Networks [104.33754950606298]
  We show how the graph structure of neural networks affect their predictive performance. A "sweet spot" of relational graphs leads to neural networks with significantly improved predictive performance. Top-performing neural networks have graph structure surprisingly similar to those of real biological neural networks.
  arXiv  Detail & Related papers  (2020-07-13T17:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.