Related papers: Designing Concise ConvNets with Columnar Stages

Designing Concise ConvNets with Columnar Stages

URL: http://arxiv.org/abs/2410.04089v1
Date: Sat, 5 Oct 2024 09:03:42 GMT
Title: Designing Concise ConvNets with Columnar Stages
Authors: Ashish Kumar, Jaesik Park,
Abstract summary: We introduce a refreshing ConvNet macro design called Columnar Stage Network (CoSNet) CoSNet has a systematically developed simple and concise structure, smaller depth, low parameter count, low FLOPs, and attention-less operations. Our evaluations show that CoSNet rivals many renowned ConvNets and Transformer designs under resource-constrained scenarios.
Score: 33.248031676529635
License: http://creativecommons.org/licenses/by/4.0/
Abstract: In the era of vision Transformers, the recent success of VanillaNet shows the huge potential of simple and concise convolutional neural networks (ConvNets). Where such models mainly focus on runtime, it is also crucial to simultaneously focus on other aspects, e.g., FLOPs, parameters, etc, to strengthen their utility further. To this end, we introduce a refreshing ConvNet macro design called Columnar Stage Network (CoSNet). CoSNet has a systematically developed simple and concise structure, smaller depth, low parameter count, low FLOPs, and attention-less operations, well suited for resource-constrained deployment. The key novelty of CoSNet is deploying parallel convolutions with fewer kernels fed by input replication, using columnar stacking of these convolutions, and minimizing the use of 1x1 convolution layers. Our comprehensive evaluations show that CoSNet rivals many renowned ConvNets and Transformer designs under resource-constrained scenarios. Code: https://github.com/ashishkumar822/CoSNet

Related papers

UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition [61.01408259741114]
We propose four architectural guidelines for designing large- Kernel-based convolutional neural networks (ConvNets) Our proposed large- Kernel-based ConvNet shows leading performance in image recognition. We discover large kernels are the key to unlocking the exceptional performance of ConvNets in domains where they were originally not proficient.
arXiv Detail & Related papers (2023-11-27T07:48:50Z)
Are Large Kernels Better Teachers than Transformers for ConvNets? [82.4742785108714]
This paper reveals a new appeal of the recently emerged large-kernel Convolutional Neural Networks (ConvNets): as the teacher in Knowledge Distillation (KD) for small- Kernel ConvNets.
arXiv Detail & Related papers (2023-05-30T21:05:23Z)
Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition [158.15602882426379]
This paper does not attempt to design a state-of-the-art method for visual recognition but investigates a more efficient way to make use of convolutions to encode spatial features. By comparing the design principles of the recent convolutional neural networks ConvNets) and Vision Transformers, we propose to simplify the self-attention by leveraging a convolutional modulation operation.
arXiv Detail & Related papers (2022-11-22T01:39:45Z)
MogaNet: Multi-order Gated Aggregation Network [64.16774341908365]
We propose a new family of modern ConvNets, dubbed MogaNet, for discriminative visual representation learning. MogaNet encapsulates conceptually simple yet effective convolutions and gated aggregation into a compact module. MogaNet exhibits great scalability, impressive efficiency of parameters, and competitive performance compared to state-of-the-art ViTs and ConvNets on ImageNet.
arXiv Detail & Related papers (2022-11-07T04:31:17Z)
ConTNet: Why not use convolution and transformer at the same time? [28.343371000297747]
We propose ConTNet, combining transformer with ConvNet architectures to provide large receptive fields. We present its superiority and effectiveness on image classification and downstream tasks. We hope that ConTNet could serve as a useful backbone for CV tasks and bring new ideas for model design.
arXiv Detail & Related papers (2021-04-27T22:29:55Z)
Capsule Network is Not More Robust than Convolutional Network [21.55939814377377]
We study the special designs in CapsNet that differ from that of a ConvNet commonly used for image classification. The study reveals that some designs, which are thought critical to CapsNet, actually can harm its robustness. We propose enhanced ConvNets simply by introducing the essential components behind the CapsNet's success.
arXiv Detail & Related papers (2021-03-29T09:47:00Z)
Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training [44.66478612082257]
Normalization techniques have become a basic component in modern convolutional neural networks (ConvNets) We introduce a simple and efficient convolutional normalization'' method that can fully exploit the convolutional structure in the Fourier domain. We show that convolutional normalization can reduce the layerwise spectral norm of the weight matrices and hence improve the Lipschitzness of the network.
arXiv Detail & Related papers (2021-03-01T00:33:04Z)
ResNet or DenseNet? Introducing Dense Shortcuts to ResNet [80.35001540483789]
This paper presents a unified perspective of dense summation to analyze them. We propose dense weighted normalized shortcuts as a solution to the dilemma between ResNet and DenseNet. Our proposed DSNet achieves significantly better results than ResNet, and achieves comparable performance as DenseNet but requiring fewer resources.
arXiv Detail & Related papers (2020-10-23T16:00:15Z)
Structured Convolutions for Efficient Neural Network Design [65.36569572213027]
We tackle model efficiency by exploiting redundancy in the textitimplicit structure of the building blocks of convolutional neural networks. We show how this decomposition can be applied to 2D and 3D kernels as well as the fully-connected layers.
arXiv Detail & Related papers (2020-08-06T04:38:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.