ResNet or DenseNet? Introducing Dense Shortcuts to ResNet

• URL: http://arxiv.org/abs/2010.12496v1
• Date: Fri, 23 Oct 2020 16:00:15 GMT
• Title: ResNet or DenseNet? Introducing Dense Shortcuts to ResNet
• Authors: Chaoning Zhang, Philipp Benz, Dawit Mureja Argaw, Seokju Lee, Junsik Kim, Francois Rameau, Jean-Charles Bazin, In So Kweon
• Abstract summary: This paper presents a unified perspective of dense summation to analyze them. We propose dense weighted normalized shortcuts as a solution to the dilemma between ResNet and DenseNet. Our proposed DSNet achieves significantly better results than ResNet, and achieves comparable performance as DenseNet but requiring fewer resources.
• Score: 80.35001540483789
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: ResNet or DenseNet? Nowadays, most deep learning based approaches are implemented with seminal backbone networks, among them the two arguably most famous ones are ResNet and DenseNet. Despite their competitive performance and overwhelming popularity, inherent drawbacks exist for both of them. For ResNet, the identity shortcut that stabilizes training also limits its representation capacity, while DenseNet has a higher capacity with multi-layer feature concatenation. However, the dense concatenation causes a new problem of requiring high GPU memory and more training time. Partially due to this, it is not a trivial choice between ResNet and DenseNet. This paper provides a unified perspective of dense summation to analyze them, which facilitates a better understanding of their core difference. We further propose dense weighted normalized shortcuts as a solution to the dilemma between them. Our proposed dense shortcut inherits the design philosophy of simple design in ResNet and DenseNet. On several benchmark datasets, the experimental results show that the proposed DSNet achieves significantly better results than ResNet, and achieves comparable performance as DenseNet but requiring fewer computation resources.

Related papers

• Designing Concise ConvNets with Columnar Stages [33.248031676529635]
  We introduce a refreshing ConvNet macro design called Columnar Stage Network (CoSNet) CoSNet has a systematically developed simple and concise structure, smaller depth, low parameter count, low FLOPs, and attention-less operations. Our evaluations show that CoSNet rivals many renowned ConvNets and Transformer designs under resource-constrained scenarios.
  arXiv  Detail & Related papers  (2024-10-05T09:03:42Z)
• MogaNet: Multi-order Gated Aggregation Network [64.16774341908365]
  We propose a new family of modern ConvNets, dubbed MogaNet, for discriminative visual representation learning. MogaNet encapsulates conceptually simple yet effective convolutions and gated aggregation into a compact module. MogaNet exhibits great scalability, impressive efficiency of parameters, and competitive performance compared to state-of-the-art ViTs and ConvNets on ImageNet.
  arXiv  Detail & Related papers  (2022-11-07T04:31:17Z)
• Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers [83.74380713308605]
  We develop a new type of transformation that is fully compatible with a variant of ReLUs -- Leaky ReLUs. We show in experiments that our method, which introduces negligible extra computational cost, validation accuracies with deep vanilla networks that are competitive with ResNets.
  arXiv  Detail & Related papers  (2022-03-15T17:49:08Z)
• ThresholdNet: Pruning Tool for Densely Connected Convolutional Networks [2.267411144256508]
  We introduce a new type of pruning tool, threshold, which refers to the principle of the threshold voltage in terms of memory. This work employs this method to connect blocks of different depths in different ways to reduce the usage of memory. Experiments show that HarDNet is twice as fast as DenseNet, and on this basis, ThresholdNet is 10% faster and 10% lower error rate than HarDNet.
  arXiv  Detail & Related papers  (2021-08-28T08:48:31Z)
• BCNet: Searching for Network Width with Bilaterally Coupled Network [56.14248440683152]
  We introduce a new supernet called Bilaterally Coupled Network (BCNet) to address this issue. In BCNet, each channel is fairly trained and responsible for the same amount of network widths, thus each network width can be evaluated more accurately. Our method achieves state-of-the-art or competing performance over other baseline methods.
  arXiv  Detail & Related papers  (2021-05-21T18:54:03Z)
• Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough [19.19644194006565]
  We show how much we can prune a neural network given a specified tolerance of accuracy drop. The proposed method has the guarantee that the discrepancy between the pruned network and the original network decays with exponentially fast rate. Empirically, our method improves prior arts on pruning various network architectures including ResNet, MobilenetV2/V3 on ImageNet.
  arXiv  Detail & Related papers  (2020-10-29T22:06:31Z)
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets [65.28292822614418]
  Giant formula for simultaneously enlarging the resolution, depth and width provides us a Rubik's cube for neural networks. This paper aims to explore the twisting rules for obtaining deep neural networks with minimum model sizes and computational costs.
  arXiv  Detail & Related papers  (2020-10-28T08:49:45Z)
• BiO-Net: Learning Recurrent Bi-directional Connections for Encoder-Decoder Architecture [82.64881585566825]
  We present a novel Bi-directional O-shape network (BiO-Net) that reuses the building blocks in a recurrent manner without introducing any extra parameters. Our method significantly outperforms the vanilla U-Net as well as other state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-01T05:07:49Z)
• DRU-net: An Efficient Deep Convolutional Neural Network for Medical Image Segmentation [2.3574651879602215]
  Residual network (ResNet) and densely connected network (DenseNet) have significantly improved the training efficiency and performance of deep convolutional neural networks (DCNNs) We propose an efficient network architecture by considering advantages of both networks.
  arXiv  Detail & Related papers  (2020-04-28T12:16:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.