WeightNet: Revisiting the Design Space of Weight Networks

• URL: http://arxiv.org/abs/2007.11823v2
• Date: Fri, 24 Jul 2020 11:47:42 GMT
• Title: WeightNet: Revisiting the Design Space of Weight Networks
• Authors: Ningning Ma, Xiangyu Zhang, Jiawei Huang, Jian Sun
• Abstract summary: We present a conceptually simple, flexible and effective framework for weight generating networks. Our approach is general that unifies two current distinct and extremely effective SENet and CondConv into the same framework on weight space.
• Score: 96.48596945711562
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a conceptually simple, flexible and effective framework for weight generating networks. Our approach is general that unifies two current distinct and extremely effective SENet and CondConv into the same framework on weight space. The method, called WeightNet, generalizes the two methods by simply adding one more grouped fully-connected layer to the attention activation layer. We use the WeightNet, composed entirely of (grouped) fully-connected layers, to directly output the convolutional weight. WeightNet is easy and memory-conserving to train, on the kernel space instead of the feature space. Because of the flexibility, our method outperforms existing approaches on both ImageNet and COCO detection tasks, achieving better Accuracy-FLOPs and Accuracy-Parameter trade-offs. The framework on the flexible weight space has the potential to further improve the performance. Code is available at https://github.com/megvii-model/WeightNet.

Related papers

• Learning to Compose SuperWeights for Neural Parameter Allocation Search [61.078949532440724]
  We show that our approach can generate parameters for many network using the same set of weights. This enables us to support tasks like efficient ensembling and anytime prediction.
  arXiv  Detail & Related papers  (2023-12-03T04:20:02Z)
• Progressive Meta-Pooling Learning for Lightweight Image Classification Model [20.076610051602618]
  We propose the Meta-Pooling framework to make the receptive field learnable for a lightweight network. We present a Progressive Meta-Pooling Learning (PMPL) strategy for the parameterized spatial enhancer to acquire a suitable receptive field size. The results on the ImageNet dataset demonstrate that MobileNetV2 using Meta-Pooling achieves top1 accuracy of 74.6%, which outperforms MobileNetV2 by 2.3%.
  arXiv  Detail & Related papers  (2023-01-24T14:28:05Z)
• Searching for Network Width with Bilaterally Coupled Network [75.43658047510334]
  We introduce a new supernet called Bilaterally Coupled Network (BCNet) to address this issue. In BCNet, each channel is fairly trained and responsible for the same amount of network widths, thus each network width can be evaluated more accurately. We propose the first open-source width benchmark on macro structures named Channel-Bench-Macro for the better comparison of width search algorithms.
  arXiv  Detail & Related papers  (2022-03-25T15:32:46Z)
• BCNet: Searching for Network Width with Bilaterally Coupled Network [56.14248440683152]
  We introduce a new supernet called Bilaterally Coupled Network (BCNet) to address this issue. In BCNet, each channel is fairly trained and responsible for the same amount of network widths, thus each network width can be evaluated more accurately. Our method achieves state-of-the-art or competing performance over other baseline methods.
  arXiv  Detail & Related papers  (2021-05-21T18:54:03Z)
• Dynamic Slimmable Network [105.74546828182834]
  We develop a dynamic network slimming regime named Dynamic Slimmable Network (DS-Net) Our DS-Net is empowered with the ability of dynamic inference by the proposed double-headed dynamic gate. It consistently outperforms its static counterparts as well as state-of-the-art static and dynamic model compression methods.
  arXiv  Detail & Related papers  (2021-03-24T15:25:20Z)
• Locally Free Weight Sharing for Network Width Search [55.155969155967284]
  Searching for network width is an effective way to slim deep neural networks with hardware budgets. We propose a locally free weight sharing strategy (CafeNet) to better evaluate each width. Our method can further boost the benchmark NAS network EfficientNet-B0 by 0.41% via searching its width more delicately.
  arXiv  Detail & Related papers  (2021-02-10T04:36:09Z)
• Sparsity-Control Ternary Weight Networks [34.00378876525579]
  We focus on training ternary weight -1, 0, +1 networks which can avoid multiplications and dramatically reduce the memory and requirements. Existing approaches to training ternary weight networks cannot control the sparsity of the ternary weights. We propose the first sparsity-control approach (SCA) to training ternary weight networks.
  arXiv  Detail & Related papers  (2020-11-01T18:06:26Z)
• Joslim: Joint Widths and Weights Optimization for Slimmable Neural Networks [37.09353669633368]
  We propose a general framework to enable joint optimization for both width configurations and weights of slimmable networks. Our framework subsumes conventional and NAS-based slimmable methods as special cases and provides flexibility to improve over existing methods. Improvements up to 1.7% and 8% in top-1 accuracy on the ImageNet dataset can be attained for MobileNetV2 considering FLOPs and memory footprint.
  arXiv  Detail & Related papers  (2020-07-23T02:05:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.