How Does Parameter Pruning Reshape DNN Representations? An Interaction-Driven Exploration

• URL: http://arxiv.org/abs/2609.06483v1
• Date: Sun, 06 Sep 2026 08:58:21 GMT
• Title: How Does Parameter Pruning Reshape DNN Representations? An Interaction-Driven Exploration
• Abstract summary: This study focuses on understanding internal factors that govern the diverse performance degradation of deep neural networks (DNNs) when different parameters are pruned.<n>In order to explain why pruning certain parameters leads to significant performance degradation but pruning other parameters does not, we examine how the pruning operation affects the interaction patterns encoded by the DNN.
• Score: 23.538451600822967
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study focuses on the scientific problem of understanding internal factors that govern the diverse performance degradation of deep neural networks (DNNs) when different parameters are pruned. In order to explain why pruning certain parameters leads to significant performance degradation but pruning other parameters does not, we examine how the pruning operation affects the interaction patterns encoded by the DNN. We find that when we progressively increase the pruning ratio, the interaction patterns encoded by DNNs exhibit a distinct three-phase dynamics, \emph{i.e.}, model performance is not largely affected until the pruning operation begins to remove low-order interactions, and low-order interactions exhibit strong generalizability. Moreover, we find that the high sensitivity of DNN performance to the pruning of certain modules is attributed to whether the pruning operation removes generalizable low-order interaction patterns.

Related papers

• Randomness of Low-Layer Parameters Determines Confusing Samples in Terms of Interaction Representations of a DNN [67.80700786901016]
  We find that the complexity of interactions encoded by a deep neural network (DNN) can explain its generalization power.<n>We also discover that the confusing samples of a DNN, which are represented by non-generalizable interactions, are determined by its low-layer parameters.
  arXiv  Detail & Related papers  (2025-02-12T18:25:13Z)
• Deep-Unrolling Multidimensional Harmonic Retrieval Algorithms on Neuromorphic Hardware [78.17783007774295]
  This paper explores the potential of conversion-based neuromorphic algorithms for highly accurate and energy-efficient single-snapshot multidimensional harmonic retrieval.<n>A novel method for converting the complex-valued convolutional layers and activations into spiking neural networks (SNNs) is developed.<n>The converted SNNs achieve almost five-fold power efficiency at moderate performance loss compared to the original CNNs.
  arXiv  Detail & Related papers  (2024-12-05T09:41:33Z)
• Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features [68.3512123520931]
  We investigate the dynamics of a deep neural network (DNN) learning interactions. In this paper, we discover the DNN learns interactions in two phases. The first phase mainly penalizes interactions of medium and high orders, and the second phase mainly learns interactions of gradually increasing orders.
  arXiv  Detail & Related papers  (2024-05-16T17:13:25Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• From Tempered to Benign Overfitting in ReLU Neural Networks [41.271773069796126]
  Overtrivialized neural networks (NNs) are observed to generalize well even when trained to perfectly fit noisy data. Recently, it was conjectured and empirically observed that the behavior of NNs is often better described as "tempered overfitting"
  arXiv  Detail & Related papers  (2023-05-24T13:36:06Z)
• Hardening DNNs against Transfer Attacks during Network Compression using Greedy Adversarial Pruning [0.1529342790344802]
  We investigate the adversarial robustness of models produced by several irregular pruning schemes and by 8-bit quantization. We find that this pruning method results in models that are resistant to transfer attacks from their uncompressed counterparts.
  arXiv  Detail & Related papers  (2022-06-15T09:13:35Z)
• Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis [94.64007376939735]
  We theoretically characterize the impact of connectivity patterns on the convergence of deep neural networks (DNNs) under gradient descent training. We show that by a simple filtration on "unpromising" connectivity patterns, we can trim down the number of models to evaluate.
  arXiv  Detail & Related papers  (2022-05-11T17:43:54Z)
• Going beyond p-convolutions to learn grayscale morphological operators [64.38361575778237]
  We present two new morphological layers based on the same principle as the p-convolutional layer. In this work, we present two new morphological layers based on the same principle as the p-convolutional layer.
  arXiv  Detail & Related papers  (2021-02-19T17:22:16Z)
• Interpreting Deep Neural Networks with Relative Sectional Propagation by Analyzing Comparative Gradients and Hostile Activations [37.11665902583138]
  We propose a new attribution method, Relative Sectional Propagation (RSP), for decomposing the output predictions of Deep Neural Networks (DNNs) We define hostile factor as an element that interferes with finding the attributions of the target and propagates it in a distinguishable way to overcome the non-suppressed nature of activated neurons. Our method makes it possible to decompose the predictions of DNNs with clearer class-discriminativeness and detailed elucidations of activation neurons compared to the conventional attribution methods.
  arXiv  Detail & Related papers  (2020-12-07T03:11:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.