Ghost-Connect Net: A Generalization-Enhanced Guidance For Sparse Deep Networks Under Distribution Shifts

• URL: http://arxiv.org/abs/2411.09199v1
• Date: Thu, 14 Nov 2024 05:43:42 GMT
• Title: Ghost-Connect Net: A Generalization-Enhanced Guidance For Sparse Deep Networks Under Distribution Shifts
• Authors: Mary Isabelle Wisell, Salimeh Yasaei Sekeh,
• Abstract summary: We introduce Ghost Connect-Net (GC-Net) to monitor the connections in the original network with distribution generalization advantage. After pruning GC-Net, the pruned locations are mapped back to the original network as pruned connections. We provide theoretical foundations for GC-Net's approach to improving generalization under distribution shifts.
• Score: 5.524804393257921
• License:
• Abstract: Sparse deep neural networks (DNNs) excel in real-world applications like robotics and computer vision, by reducing computational demands that hinder usability. However, recent studies aim to boost DNN efficiency by trimming redundant neurons or filters based on task relevance, but neglect their adaptability to distribution shifts. We aim to enhance these existing techniques by introducing a companion network, Ghost Connect-Net (GC-Net), to monitor the connections in the original network with distribution generalization advantage. GC-Net's weights represent connectivity measurements between consecutive layers of the original network. After pruning GC-Net, the pruned locations are mapped back to the original network as pruned connections, allowing for the combination of magnitude and connectivity-based pruning methods. Experimental results using common DNN benchmarks, such as CIFAR-10, Fashion MNIST, and Tiny ImageNet show promising results for hybridizing the method, and using GC-Net guidance for later layers of a network and direct pruning on earlier layers. We provide theoretical foundations for GC-Net's approach to improving generalization under distribution shifts.

Related papers

• Learning Load Balancing with GNN in MPTCP-Enabled Heterogeneous Networks [13.178956651532213]
  We propose a graph neural network (GNN)-based model to tackle the LB problem for MP TCP-enabled HetNets. Compared to the conventional deep neural network (DNN), the proposed GNN-based model exhibits two key strengths.
  arXiv  Detail & Related papers  (2024-10-22T15:49:53Z)
• SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models [19.479746878680707]
  Layer pruning is a potent approach to reduce network size and improve computational efficiency. We propose a Similarity Guided fast Layer Partition pruning for compressing large deep models. Our method outperforms the state-of-the-art methods in both accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2024-10-14T04:01:08Z)
• Searching for Network Width with Bilaterally Coupled Network [75.43658047510334]
  We introduce a new supernet called Bilaterally Coupled Network (BCNet) to address this issue. In BCNet, each channel is fairly trained and responsible for the same amount of network widths, thus each network width can be evaluated more accurately. We propose the first open-source width benchmark on macro structures named Channel-Bench-Macro for the better comparison of width search algorithms.
  arXiv  Detail & Related papers  (2022-03-25T15:32:46Z)
• Edge Rewiring Goes Neural: Boosting Network Resilience via Policy Gradient [62.660451283548724]
  ResiNet is a reinforcement learning framework to discover resilient network topologies against various disasters and attacks. We show that ResiNet achieves a near-optimal resilience gain on multiple graphs while balancing the utility, with a large margin compared to existing approaches.
  arXiv  Detail & Related papers  (2021-10-18T06:14:28Z)
• BCNet: Searching for Network Width with Bilaterally Coupled Network [56.14248440683152]
  We introduce a new supernet called Bilaterally Coupled Network (BCNet) to address this issue. In BCNet, each channel is fairly trained and responsible for the same amount of network widths, thus each network width can be evaluated more accurately. Our method achieves state-of-the-art or competing performance over other baseline methods.
  arXiv  Detail & Related papers  (2021-05-21T18:54:03Z)
• Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks [78.65792427542672]
  Dynamic Graph Network (DG-Net) is a complete directed acyclic graph, where the nodes represent convolutional blocks and the edges represent connection paths. Instead of using the same path of the network, DG-Net aggregates features dynamically in each node, which allows the network to have more representation ability.
  arXiv  Detail & Related papers  (2020-10-02T16:50:26Z)
• Compact Neural Representation Using Attentive Network Pruning [1.0152838128195465]
  We describe a Top-Down attention mechanism that is added to a Bottom-Up feedforward network to select important connections and subsequently prune redundant ones at all parametric layers. Our method not only introduces a novel hierarchical selection mechanism as the basis of pruning but also remains competitive with previous baseline methods in the experimental evaluation.
  arXiv  Detail & Related papers  (2020-05-10T03:20:01Z)
• DRU-net: An Efficient Deep Convolutional Neural Network for Medical Image Segmentation [2.3574651879602215]
  Residual network (ResNet) and densely connected network (DenseNet) have significantly improved the training efficiency and performance of deep convolutional neural networks (DCNNs) We propose an efficient network architecture by considering advantages of both networks.
  arXiv  Detail & Related papers  (2020-04-28T12:16:24Z)
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio [101.84651388520584]
  This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs. Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-06T15:51:00Z)
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions [76.05981545084738]
  We propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost. We first construct a baseline network by modifying and binarizing a compact real-valued network with parameter-free shortcuts. We show that the proposed ReActNet outperforms all the state-of-the-arts by a large margin.
  arXiv  Detail & Related papers  (2020-03-07T02:12:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.