GradAttn: Replacing Fixed Residual Connections with Task-Modulated Attention Pathways

• URL: http://arxiv.org/abs/2603.26756v1
• Date: Mon, 23 Mar 2026 14:45:07 GMT
• Title: GradAttn: Replacing Fixed Residual Connections with Task-Modulated Attention Pathways
• Abstract summary: GradAttn is a CNN-transformer framework that replaces fixed residual connections with attention-controlled gradient flow.<n>For representational analysis, we evaluated three GradAttn variants across eight diverse datasets.<n>Results demonstrate that GradAttn outperforms ResNet-18 on five of eight datasets.
• Score: 2.105564340986074
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep ConvNets suffer from gradient signal degradation as network depth increases, limiting effective feature learning in complex architectures. ResNet addressed this through residual connections, but these fixed short-circuits cannot adapt to varying input complexity or selectively emphasize task relevant features across network hierarchies. This study introduces GradAttn, a hybrid CNN-transformer framework that replaces fixed residual connections with attention-controlled gradient flow. By extracting multi-scale CNN features at different depths and regulating them through self-attention, GradAttn dynamically weights shallow texture features and deep semantic representations. For representational analysis, we evaluated three GradAttn variants across eight diverse datasets, from natural images, medical imaging, to fashion recognition. Results demonstrate that GradAttn outperforms ResNet-18 on five of eight datasets, achieving up to +11.07% accuracy improvement on FashionMNIST while maintaining comparable network size. Gradient flow analysis reveals that controlled instabilities, introduced by attention, often coincide with improved generalization, challenging the assumption that perfect stability is optimal. Furthermore, positional encoding effectiveness proves dataset dependent, with CNN hierarchies frequently encoding sufficient spatial structure. These findings allow attention mechanisms as enablers of learnable gradient control, offering a new paradigm for adaptive representation learning in deep neural architectures.

Related papers

• Auto-Compressing Networks [51.221103189527014]
  We introduce Auto-compression Networks (ACNs), an architectural variant where long feedforward connections from each layer replace traditional short residual connections.<n>We show that ACNs exhibit enhanced noise compared to residual networks, superior performance in low-data settings, and mitigate catastrophic forgetting.<n>These findings establish ACNs as a practical approach to developing efficient neural architectures.
  arXiv  Detail & Related papers  (2025-06-11T13:26:09Z)
• Pruning Deep Convolutional Neural Network Using Conditional Mutual Information [10.302118493842647]
  Convolutional Neural Networks (CNNs) achieve high performance in image classification tasks but are challenging to deploy on resource-limited hardware.<n>We propose a structured filter-pruning approach for CNNs that identifies and selectively retains the most informative features in each layer.
  arXiv  Detail & Related papers  (2024-11-27T18:23:59Z)
• Distance Weighted Trans Network for Image Completion [52.318730994423106]
  We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components. CNNs are used to augment the local texture information of coarse priors. DWT blocks are used to recover certain coarse textures and coherent visual structures.
  arXiv  Detail & Related papers  (2023-10-11T12:46:11Z)
• Adaptive Growth: Real-time CNN Layer Expansion [0.0]
  This research presents a new algorithm that allows the convolutional layer of a Convolutional Neural Network (CNN) to dynamically evolve based on data input. Instead of a rigid architecture, our approach iteratively introduces kernels to the convolutional layer, gauging its real-time response to varying data. Remarkably, our unsupervised method has outstripped its supervised counterparts across diverse datasets.
  arXiv  Detail & Related papers  (2023-09-06T14:43:58Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)
• Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis [94.64007376939735]
  We theoretically characterize the impact of connectivity patterns on the convergence of deep neural networks (DNNs) under gradient descent training. We show that by a simple filtration on "unpromising" connectivity patterns, we can trim down the number of models to evaluate.
  arXiv  Detail & Related papers  (2022-05-11T17:43:54Z)
• SIRe-Networks: Skip Connections over Interlaced Multi-Task Learning and Residual Connections for Structure Preserving Object Classification [28.02302915971059]
  In this paper, we introduce an interlaced multi-task learning strategy, defined SIRe, to reduce the vanishing gradient in relation to the object classification task. The presented methodology directly improves a convolutional neural network (CNN) by enforcing the input image structure preservation through auto-encoders. To validate the presented methodology, a simple CNN and various implementations of famous networks are extended via the SIRe strategy and extensively tested on the CIFAR100 dataset.
  arXiv  Detail & Related papers  (2021-10-06T13:54:49Z)
• Learning Deep Interleaved Networks with Asymmetric Co-Attention for Image Restoration [65.11022516031463]
  We present a deep interleaved network (DIN) that learns how information at different states should be combined for high-quality (HQ) images reconstruction. In this paper, we propose asymmetric co-attention (AsyCA) which is attached at each interleaved node to model the feature dependencies. Our presented DIN can be trained end-to-end and applied to various image restoration tasks.
  arXiv  Detail & Related papers  (2020-10-29T15:32:00Z)
• Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks [78.65792427542672]
  Dynamic Graph Network (DG-Net) is a complete directed acyclic graph, where the nodes represent convolutional blocks and the edges represent connection paths. Instead of using the same path of the network, DG-Net aggregates features dynamically in each node, which allows the network to have more representation ability.
  arXiv  Detail & Related papers  (2020-10-02T16:50:26Z)
• Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives [73.15276998621582]
  We propose a generic feature learning mechanism to advance CNN training with enhanced generalization ability. Partially inspired by DSN, we fork delicately designed side branches from the intermediate layers of a given neural network. Experiments on both category and instance recognition tasks demonstrate the substantial improvements of our proposed method.
  arXiv  Detail & Related papers  (2020-03-24T09:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.