Online Convolutional Re-parameterization

• URL: http://arxiv.org/abs/2204.00826v1
• Date: Sat, 2 Apr 2022 09:50:19 GMT
• Title: Online Convolutional Re-parameterization
• Authors: Mu Hu, Junyi Feng, Jiashen Hua, Baisheng Lai, Jianqiang Huang, Xiaojin Gong, Xiansheng Hua
• Abstract summary: We present online convolutional re- parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks.
• Score: 51.97831675242173
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Structural re-parameterization has drawn increasing attention in various computer vision tasks. It aims at improving the performance of deep models without introducing any inference-time cost. Though efficient during inference, such models rely heavily on the complicated training-time blocks to achieve high accuracy, leading to large extra training cost. In this paper, we present online convolutional re-parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. To achieve this goal, we introduce a linear scaling layer for better optimizing the online blocks. Assisted with the reduced training cost, we also explore some more effective re-param components. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. Meanwhile, equipped with OREPA, the models outperform previous methods on ImageNet by up to +0.6%.We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks. Codes are available at https://github.com/JUGGHM/OREPA_CVPR2022 .

Related papers

• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Time-, Memory- and Parameter-Efficient Visual Adaptation [75.28557015773217]
  We propose an adaptation method which does not backpropagate gradients through the backbone. We achieve this by designing a lightweight network in parallel that operates on features from the frozen, pretrained backbone.
  arXiv  Detail & Related papers  (2024-02-05T10:55:47Z)
• Always-Sparse Training by Growing Connections with Guided Stochastic Exploration [46.4179239171213]
  We propose an efficient always-sparse training algorithm with excellent scaling to larger and sparser models. We evaluate our method on CIFAR-10/100 and ImageNet using VGG, and ViT models, and compare it against a range of sparsification methods.
  arXiv  Detail & Related papers  (2024-01-12T21:32:04Z)
• Surrogate Lagrangian Relaxation: A Path To Retrain-free Deep Neural Network Pruning [9.33753001494221]
  Network pruning is a widely used technique to reduce computation cost and model size for deep neural networks. In this paper, we develop a systematic weight-pruning optimization approach based on Surrogate Lagrangian relaxation.
  arXiv  Detail & Related papers  (2023-04-08T22:48:30Z)
• A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustness [11.35810118757863]
  Existing models that achieve state-of-the-art (SOTA) performance on both clean and adversarially-perturbed images rely on convolution operations conditioned with feature-wise linear modulation (FiLM) layers. We present a fast learnable once-for-all adversarial training (FLOAT) algorithm, which instead of the existing FiLM-based conditioning, presents a unique weight conditioned learning that requires no additional layer. In particular, we add scaled noise to the weight tensors that enables a trade-off between clean and adversarial performance.
  arXiv  Detail & Related papers  (2022-03-28T19:25:36Z)
• Recursive Least-Squares Estimator-Aided Online Learning for Visual Tracking [58.14267480293575]
  We propose a simple yet effective online learning approach for few-shot online adaptation without requiring offline training. It allows an in-built memory retention mechanism for the model to remember the knowledge about the object seen before. We evaluate our approach based on two networks in the online learning families for tracking, i.e., multi-layer perceptrons in RT-MDNet and convolutional neural networks in DiMP.
  arXiv  Detail & Related papers  (2021-12-28T06:51:18Z)
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings [89.63764845984076]
  We present Stored Embeddings for Efficient Reinforcement Learning (SEER) SEER is a simple modification of existing off-policy deep reinforcement learning methods. We show that SEER does not degrade the performance of RLizable agents while significantly saving computation and memory.
  arXiv  Detail & Related papers  (2021-03-04T08:14:10Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.