ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling

• URL: http://arxiv.org/abs/2602.09009v1
• Date: Mon, 09 Feb 2026 18:54:18 GMT
• Title: ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
• Authors: Yilang Zhang, Bingcong Li, Niao He, Georgios B. Giannakis,
• Abstract summary: Scaling network depth has been a central driver behind the success of modern foundation models.<n>This paper revisits the default mechanism for deepening neural networks, namely residual connections.<n>We introduce adaptive neural connection reassignment (ANCRe), a principled and lightweight framework that parameterizes and learns residual connectivities from the data.
• Score: 57.91760520589592
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scaling network depth has been a central driver behind the success of modern foundation models, yet recent investigations suggest that deep layers are often underutilized. This paper revisits the default mechanism for deepening neural networks, namely residual connections, from an optimization perspective. Rigorous analysis proves that the layout of residual connections can fundamentally shape convergence behavior, and even induces an exponential gap in convergence rates. Prompted by this insight, we introduce adaptive neural connection reassignment (ANCRe), a principled and lightweight framework that parameterizes and learns residual connectivities from the data. ANCRe adaptively reassigns residual connections with negligible computational and memory overhead ($<1\%$), while enabling more effective utilization of network depth. Extensive numerical tests across pre-training of large language models, diffusion models, and deep ResNets demonstrate consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.

Related papers

• Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning [57.3885832382455]
  We show that introducing static network sparsity alone can unlock further scaling potential beyond dense counterparts with state-of-the-art architectures.<n>Our analysis reveals that, in contrast to naively scaling up dense DRL networks, such sparse networks achieve both higher parameter efficiency for network expressivity.
  arXiv  Detail & Related papers  (2025-06-20T17:54:24Z)
• Lattice-Based Pruning in Recurrent Neural Networks via Poset Modeling [0.0]
  Recurrent neural networks (RNNs) are central to sequence modeling tasks, yet their high computational complexity poses challenges for scalability and real-time deployment.<n>We introduce a novel framework that models RNNs as partially ordered sets (posets) and constructs corresponding dependency lattices.<n>By identifying meet irreducible neurons, our lattice-based pruning algorithm selectively retains critical connections while eliminating redundant ones.
  arXiv  Detail & Related papers  (2025-02-23T10:11:38Z)
• Efficient Training of Deep Neural Operator Networks via Randomized Sampling [0.0]
  We introduce a random sampling technique to be adopted the training of DeepONet.<n>We demonstrate substantial reductions in training time while achieving comparable or lower overall test errors relative to the traditional training approach.<n>Our results indicate that incorporating randomization in the trunk network inputs during training enhances the efficiency and robustness of DeepONet.
  arXiv  Detail & Related papers  (2024-09-20T07:18:31Z)
• Imbedding Deep Neural Networks [0.0]
  Continuous depth neural networks, such as Neural ODEs, have refashioned the understanding of residual neural networks in terms of non-linear vector-valued optimal control problems. We propose a new approach which explicates the network's depth' as a fundamental variable, thus reducing the problem to a system of forward-facing initial value problems.
  arXiv  Detail & Related papers  (2022-01-31T22:00:41Z)
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks [56.084798078072396]
  We take a step towards closing the gap between theory and practice by significantly improving the known theoretical bounds on both the network width and the convergence time. We show that convergence to a global minimum is guaranteed for networks with quadratic widths in the sample size and linear in their depth at a time logarithmic in both. Our analysis and convergence bounds are derived via the construction of a surrogate network with fixed activation patterns that can be transformed at any time to an equivalent ReLU network of a reasonable size.
  arXiv  Detail & Related papers  (2021-01-12T00:40:45Z)
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning [97.28695683236981]
  More gradient updates decrease the expressivity of the current value network. We demonstrate this phenomenon on Atari and Gym benchmarks, in both offline and online RL settings.
  arXiv  Detail & Related papers  (2020-10-27T17:55:16Z)
• Kernel-Based Smoothness Analysis of Residual Networks [85.20737467304994]
  Residual networks (ResNets) stand out among these powerful modern architectures. In this paper, we show another distinction between the two models, namely, a tendency of ResNets to promote smoothers than gradients.
  arXiv  Detail & Related papers  (2020-09-21T16:32:04Z)
• Iterative Network for Image Super-Resolution [69.07361550998318]
  Single image super-resolution (SISR) has been greatly revitalized by the recent development of convolutional neural networks (CNN) This paper provides a new insight on conventional SISR algorithm, and proposes a substantially different approach relying on the iterative optimization. A novel iterative super-resolution network (ISRN) is proposed on top of the iterative optimization.
  arXiv  Detail & Related papers  (2020-05-20T11:11:47Z)
• Compact Neural Representation Using Attentive Network Pruning [1.0152838128195465]
  We describe a Top-Down attention mechanism that is added to a Bottom-Up feedforward network to select important connections and subsequently prune redundant ones at all parametric layers. Our method not only introduces a novel hierarchical selection mechanism as the basis of pruning but also remains competitive with previous baseline methods in the experimental evaluation.
  arXiv  Detail & Related papers  (2020-05-10T03:20:01Z)
• Depth Enables Long-Term Memory for Recurrent Neural Networks [0.0]
  We introduce a measure of the network's ability to support information flow across time, referred to as the Start-End separation rank. We prove that deep recurrent networks support Start-End separation ranks which are higher than those supported by their shallow counterparts.
  arXiv  Detail & Related papers  (2020-03-23T10:29:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.