Deeply Shared Filter Bases for Parameter-Efficient Convolutional Neural Networks

• URL: http://arxiv.org/abs/2006.05066v4
• Date: Sun, 21 Nov 2021 09:53:04 GMT
• Title: Deeply Shared Filter Bases for Parameter-Efficient Convolutional Neural Networks
• Abstract summary: A modern convolutional neural networks (CNNs) have massive identical convolution blocks. In this paper, we show how a filter basis is separated and learned while effectively avoiding the vanishing/exploding gradients problem during training. Experimental results on image classification and object detection show that our approach, unlike previous parameter-sharing approaches, does not trade performance to save parameters.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern convolutional neural networks (CNNs) have massive identical convolution blocks, and, hence, recursive sharing of parameters across these blocks has been proposed to reduce the amount of parameters. However, naive sharing of parameters poses many challenges such as limited representational power and the vanishing/exploding gradients problem of recursively shared parameters. In this paper, we present a recursive convolution block design and training method, in which a recursively shareable part, or a filter basis, is separated and learned while effectively avoiding the vanishing/exploding gradients problem during training. We show that the unwieldy vanishing/exploding gradients problem can be controlled by enforcing the elements of the filter basis orthonormal, and empirically demonstrate that the proposed orthogonality regularization improves the flow of gradients during training. Experimental results on image classification and object detection show that our approach, unlike previous parameter-sharing approaches, does not trade performance to save parameters and consistently outperforms overparameterized counterpart networks. This superior performance demonstrates that the proposed recursive convolution block design and the orthogonality regularization not only prevent performance degradation, but also consistently improve the representation capability while a significant amount of parameters are recursively shared.

Related papers

• Recovery Guarantees of Unsupervised Neural Networks for Inverse Problems trained with Gradient Descent [0.6522338519818377]
  We show that convergence and recovery guarantees for generic loss functions hold true when trained through gradient flow. We also show that the discretization only affects the overparametrization bound for a two-layer DIP network by a constant.
  arXiv  Detail & Related papers  (2024-03-08T15:45:13Z)
• Robust Stochastically-Descending Unrolled Networks [85.6993263983062]
  Deep unrolling is an emerging learning-to-optimize method that unrolls a truncated iterative algorithm in the layers of a trainable neural network. We show that convergence guarantees and generalizability of the unrolled networks are still open theoretical problems. We numerically assess unrolled architectures trained under the proposed constraints in two different applications.
  arXiv  Detail & Related papers  (2023-12-25T18:51:23Z)
• Unnatural Algorithms in Machine Learning [0.0]
  We show that optimization algorithms with this property can be viewed as discrete approximations of natural gradient descent. We introduce a simple method of introducing this naturality more generally and examine a number of popular machine learning training algorithms.
  arXiv  Detail & Related papers  (2023-12-07T22:43:37Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks [82.03459331544737]
  We study the performance of ConvResNeXts, trained with weight decay from the perspective of nonparametric classification. Our analysis allows for infinitely many building blocks in ConvResNeXts, and shows that weight decay implicitly enforces sparsity on these blocks.
  arXiv  Detail & Related papers  (2023-07-04T11:08:03Z)
• Contrastive inverse regression for dimension reduction [0.0]
  We propose a supervised dimension reduction method called contrastive inverse regression (CIR) specifically designed for the contrastive setting. CIR introduces an optimization problem defined on the Stiefel manifold with a non-standard loss function. We prove the convergence of CIR to a local optimum using a gradient descent-based algorithm, and our numerical study empirically demonstrates the improved performance over competing methods for high-dimensional data.
  arXiv  Detail & Related papers  (2023-05-20T21:44:11Z)
• Hybrid ISTA: Unfolding ISTA With Convergence Guarantees Using Free-Form Deep Neural Networks [50.193061099112626]
  It is promising to solve linear inverse problems by unfolding iterative algorithms as deep neural networks (DNNs) with learnable parameters. Existing ISTA-based unfolded algorithms restrict the network architectures for iterative updates with the partial weight coupling structure to guarantee convergence. This paper is the first to provide a convergence-provable framework that enables free-form DNNs in ISTA-based unfolded algorithms.
  arXiv  Detail & Related papers  (2022-04-25T13:17:57Z)
• Boosting Pruned Networks with Linear Over-parameterization [8.796518772724955]
  Structured pruning compresses neural networks by reducing channels (filters) for fast inference and low footprint at run-time. To restore accuracy after pruning, fine-tuning is usually applied to pruned networks. We propose a novel method that first linearly over- parameterizes the compact layers in pruned networks to enlarge the number of fine-tuning parameters.
  arXiv  Detail & Related papers  (2022-04-25T05:30:26Z)
• Backpropagation-Free Learning Method for Correlated Fuzzy Neural Networks [2.1320960069210475]
  This paper proposes a novel stepwise learning approach based on estimating desired premise parts' outputs. It does not require backpropagating the output error to learn the premise parts' parameters. It is successfully applied to real-world time-series prediction and regression problems.
  arXiv  Detail & Related papers  (2020-11-25T20:56:05Z)
• Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks [107.77595511218429]
  In this paper, we investigate the empirical Rademacher complexity related to intermediate layers of deep neural networks. We propose a feature distortion method (Disout) for addressing the aforementioned problem. The superiority of the proposed feature map distortion for producing deep neural network with higher testing performance is analyzed and demonstrated.
  arXiv  Detail & Related papers  (2020-02-23T13:59:13Z)
• Supervised Learning for Non-Sequential Data: A Canonical Polyadic Decomposition Approach [85.12934750565971]
  Efficient modelling of feature interactions underpins supervised learning for non-sequential tasks. To alleviate this issue, it has been proposed to implicitly represent the model parameters as a tensor. For enhanced expressiveness, we generalize the framework to allow feature mapping to arbitrarily high-dimensional feature vectors.
  arXiv  Detail & Related papers  (2020-01-27T22:38:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.