Alternating Gradient Flow Utility: A Unified Metric for Structural Pruning and Dynamic Routing in Deep Networks

• URL: http://arxiv.org/abs/2603.12354v2
• Date: Tue, 17 Mar 2026 07:35:13 GMT
• Title: Alternating Gradient Flow Utility: A Unified Metric for Structural Pruning and Dynamic Routing in Deep Networks
• Abstract summary: We propose a decoupled kinetic paradigm inspired by Alternating Gradient Flow (AGF)<n>AGF accurately captures the network's structural "kinetic utility"<n>We design a hybrid routing framework that decouples AGF-guided offline structural search from online execution via zero-cost physical priors.
• Score: 52.153950303594684
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Efficient deep learning traditionally relies on static heuristics like weight magnitude or activation awareness (e.g., Wanda, RIA). While successful in unstructured settings, we observe a critical limitation when applying these metrics to the structural pruning of deep vision networks. These contemporary metrics suffer from a magnitude bias, failing to preserve critical functional pathways. To overcome this, we propose a decoupled kinetic paradigm inspired by Alternating Gradient Flow (AGF), utilizing an absolute feature-space Taylor expansion to accurately capture the network's structural "kinetic utility". First, we uncover a topological phase transition at extreme sparsity, where AGF successfully preserves baseline functionality and exhibits topological implicit regularization, avoiding the collapse seen in models trained from scratch. Second, transitioning to architectures without strict structural priors, we reveal a phenomenon of Sparsity Bottleneck in Vision Transformers (ViTs). Through a gradient-magnitude decoupling analysis, we discover that dynamic signals suffer from signal compression in converged models, rendering them suboptimal for real-time routing. Finally, driven by these empirical constraints, we design a hybrid routing framework that decouples AGF-guided offline structural search from online execution via zero-cost physical priors. We validate our paradigm on large-scale benchmarks: under a 75% compression stress test on ImageNet-1K, AGF effectively avoids the structural collapse where traditional metrics aggressively fall below random sampling. Furthermore, when systematically deployed for dynamic inference on ImageNet-100, our hybrid approach achieves Pareto-optimal efficiency. It reduces the usage of the heavy expert by approximately 50% (achieving an estimated overall cost of 0.92$\times$) without sacrificing the full-model accuracy.

Related papers

• When GNNs Fail: Quantifying and Overcoming Temporal Correlation Volatility in Time Series [14.456983509027195]
  We propose Graph Layer for Inference in Dynamic En- vironments (GLIDE), a novel GNN layer enhanced by two theoretically grounded design mechanisms.<n>Experiments on synthetic and real-world benchmarks show that GLIDE improves average performance by up to 45.6% across static and dynamic settings.
  arXiv  Detail & Related papers  (2026-08-07T15:28:45Z)
• Relative Kinetic Utility for Reasoning-Aware Structural Pruning in Large Language Models [0.0]
  We propose a novel framework that elevates discrete pruning to a continuous kinetic integral over the depth manifold of a model.<n>Experiments on Qwen-2.5-7B and LLaMA-3-8B improve performance in the high-sparsity regime around 40%.<n>RKU attains 13.34% accuracy on GSM8K at 40% sparsity, outperforming the strongest baseline.
  arXiv  Detail & Related papers  (2026-05-09T15:47:53Z)
• BOOST-RPF: Boosted Sequential Trees for Radial Power Flow [0.0]
  We introduce BOOST-RPF, a novel method that reformulates voltage prediction from a global graph regression task into a sequential path-based learning problem.<n>We evaluate three architectural variants: Absolute Voltage, Parent Residual, and Physics-Informed Residual.<n> Benchmarked against the Kerber Dorfnetz grid and the ENGAGE suite, BOOST-RPF achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2026-03-23T13:41:36Z)
• StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models [98.72926158261937]
  We propose a training-free token pruning framework for Visual AutoRegressive models.<n>We employ a lightweight high-pass filter to capture local texture details, while leveraging Principal Component Analysis (PCA) to preserve global structural information.<n>To maintain valid next-scale prediction under sparse tokens, we introduce a nearest neighbor feature propagation strategy.
  arXiv  Detail & Related papers  (2026-03-02T11:35:05Z)
• SpanNorm: Reconciling Training Stability and Performance in Deep Transformers [55.100133502295996]
  We propose SpanNorm, a novel technique designed to resolve the dilemma by integrating the strengths of both paradigms.<n>We provide a theoretical analysis demonstrating that SpanNorm, combined with a principled scaling strategy, maintains bounded signal variance throughout the network.<n> Empirically, SpanNorm consistently outperforms standard normalization schemes in both dense and Mixture-of-Experts (MoE) scenarios.
  arXiv  Detail & Related papers  (2026-01-30T05:21:57Z)
• TGSBM: Transformer-Guided Stochastic Block Model for Link Prediction [13.840265247620556]
  Link prediction is a cornerstone of the Web ecosystem, powering applications from recommendation and search to knowledge graph completion and collaboration forecasting.<n>Existing approaches face notable limitations: traditional graph neural networks struggle to capture global dependencies, while recent graph transformers achieve strong performance but incur lack of interpretable structural structure.<n>We propose text-Guided Block Model, a framework that integrates the principled generative structure of Overlapping Block Models with the power of sparse Graph Transformers.
  arXiv  Detail & Related papers  (2026-01-28T14:32:24Z)
• Rethinking Autoregressive Models for Lossless Image Compression via Hierarchical Parallelism and Progressive Adaptation [75.58269386927076]
  Autoregressive (AR) models are often dismissed as impractical due to prohibitive computational cost.<n>This work re-thinks this paradigm, introducing a framework built on hierarchical parallelism and progressive adaptation.<n> Experiments on diverse datasets (natural, satellite, medical) validate that our method achieves new state-of-the-art compression.
  arXiv  Detail & Related papers  (2025-11-14T06:27:58Z)
• Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis [5.486667906157719]
  Eigen Neural Network (ENN) is a novel architecture that re parameterizes each layer's weights in a layer-shared, learned orthonormal eigenbasis.<n>When integrated with standard BP, ENN consistently outperforms state-of-the-art methods on large-scale image classification benchmarks.
  arXiv  Detail & Related papers  (2025-08-02T06:33:58Z)
• Physics-Informed Graph Neural Networks for Transverse Momentum Estimation in CMS Trigger Systems [0.0]
  Real-time particle transverse momentum ($p_T$) estimation in high-energy physics demands efficient algorithms under strict hardware constraints.<n>We propose a physics-informed graph neural network (GNN) framework that systematically encodes detector geometry and physical observables.<n>Our co-design methodology yields superior accuracy-efficiency trade-offs compared to existing baselines.
  arXiv  Detail & Related papers  (2025-07-25T12:19:57Z)
• (PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetwork [60.889175951038496]
  Large-scale neural networks have demonstrated remarkable performance in different domains like vision and language processing. One of the key questions of structural pruning is how to estimate the channel significance. We propose a novel algorithmic framework, namely textttPASS. It is a tailored hyper-network to take both visual prompts and network weight statistics as input, and output layer-wise channel sparsity in a recurrent manner.
  arXiv  Detail & Related papers  (2024-07-24T16:47:45Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• On skip connections and normalisation layers in deep optimisation [32.51139594406463]
  We introduce a general theoretical framework for the study of optimisation of deep neural networks. Our framework determines the curvature and regularity properties of multilayer loss landscapes. We identify a novel causal mechanism by which skip connections accelerate training.
  arXiv  Detail & Related papers  (2022-10-10T06:22:46Z)
• Graph-based Algorithm Unfolding for Energy-aware Power Allocation in Wireless Networks [27.600081147252155]
  We develop a novel graph sumable framework to maximize energy efficiency in wireless communication networks. We show the permutation training which is a desirable property for models of wireless network data. Results demonstrate its generalizability across different network topologies.
  arXiv  Detail & Related papers  (2022-01-27T20:23:24Z)
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search [55.164053971213576]
  convolutional neural network has achieved great success in fulfilling computer vision tasks despite large computation overhead. Structured (channel) pruning is usually applied to reduce the model redundancy while preserving the network structure. Existing structured pruning methods require hand-crafted rules which may lead to tremendous pruning space.
  arXiv  Detail & Related papers  (2020-11-04T07:43:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.