Enhancing Convolutional Neural Networks with Higher-Order Numerical Difference Methods

• URL: http://arxiv.org/abs/2409.04977v1
• Date: Sun, 8 Sep 2024 05:13:58 GMT
• Title: Enhancing Convolutional Neural Networks with Higher-Order Numerical Difference Methods
• Authors: Qi Wang, Zijun Gao, Mingxiu Sui, Taiyuan Mei, Xiaohan Cheng, Iris Li,
• Abstract summary: Convolutional Neural Networks (CNNs) have been able to assist humans in solving many real-world problems. This paper proposes a stacking scheme based on the linear multi-step method to enhance the performance of CNNs.
• Score: 6.26650196870495
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rise of deep learning technology in practical applications, Convolutional Neural Networks (CNNs) have been able to assist humans in solving many real-world problems. To enhance the performance of CNNs, numerous network architectures have been explored. Some of these architectures are designed based on the accumulated experience of researchers over time, while others are designed through neural architecture search methods. The improvements made to CNNs by the aforementioned methods are quite significant, but most of the improvement methods are limited in reality by model size and environmental constraints, making it difficult to fully realize the improved performance. In recent years, research has found that many CNN structures can be explained by the discretization of ordinary differential equations. This implies that we can design theoretically supported deep network structures using higher-order numerical difference methods. It should be noted that most of the previous CNN model structures are based on low-order numerical methods. Therefore, considering that the accuracy of linear multi-step numerical difference methods is higher than that of the forward Euler method, this paper proposes a stacking scheme based on the linear multi-step method. This scheme enhances the performance of ResNet without increasing the model size and compares it with the Runge-Kutta scheme. The experimental results show that the performance of the stacking scheme proposed in this paper is superior to existing stacking schemes (ResNet and HO-ResNet), and it has the capability to be extended to other types of neural networks.

Related papers

• Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
  This paper introduces the SANE approach to weight-space learning. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
  arXiv  Detail & Related papers  (2024-06-14T13:12:07Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• Receptive Field Refinement for Convolutional Neural Networks Reliably Improves Predictive Performance [1.52292571922932]
  We present a new approach to receptive field analysis that can yield these types of theoretical and empirical performance gains. Our approach is able to improve ImageNet1K performance across a wide range of well-known, state-of-the-art (SOTA) model classes.
  arXiv  Detail & Related papers  (2022-11-26T05:27:44Z)
• Pushing the Efficiency Limit Using Structured Sparse Convolutions [82.31130122200578]
  We propose Structured Sparse Convolution (SSC), which leverages the inherent structure in images to reduce the parameters in the convolutional filter. We show that SSC is a generalization of commonly used layers (depthwise, groupwise and pointwise convolution) in efficient architectures'' Architectures based on SSC achieve state-of-the-art performance compared to baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet classification benchmarks.
  arXiv  Detail & Related papers  (2022-10-23T18:37:22Z)
• SIRe-Networks: Skip Connections over Interlaced Multi-Task Learning and Residual Connections for Structure Preserving Object Classification [28.02302915971059]
  In this paper, we introduce an interlaced multi-task learning strategy, defined SIRe, to reduce the vanishing gradient in relation to the object classification task. The presented methodology directly improves a convolutional neural network (CNN) by enforcing the input image structure preservation through auto-encoders. To validate the presented methodology, a simple CNN and various implementations of famous networks are extended via the SIRe strategy and extensively tested on the CIFAR100 dataset.
  arXiv  Detail & Related papers  (2021-10-06T13:54:49Z)
• D-DARTS: Distributed Differentiable Architecture Search [75.12821786565318]
  Differentiable ARchiTecture Search (DARTS) is one of the most trending Neural Architecture Search (NAS) methods. We propose D-DARTS, a novel solution that addresses this problem by nesting several neural networks at cell-level.
  arXiv  Detail & Related papers  (2021-08-20T09:07:01Z)
• Connections between Numerical Algorithms for PDEs and Neural Networks [8.660429288575369]
  We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural networks. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks.
  arXiv  Detail & Related papers  (2021-07-30T16:42:45Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Genetic U-Net: Automatically Designed Deep Networks for Retinal Vessel Segmentation Using a Genetic Algorithm [2.6629444004809826]
  Genetic U-Net is proposed to generate a U-shaped convolutional neural network (CNN) that can achieve better retinal vessel segmentation but with fewer architecture-based parameters. The experimental results show that the architecture obtained using the proposed method offered a superior performance with less than 1% of the number of the original U-Net parameters in particular.
  arXiv  Detail & Related papers  (2020-10-29T13:31:36Z)
• Learned Greedy Method (LGM): A Novel Neural Architecture for Sparse Coding and Beyond [24.160276545294288]
  We propose an unfolded version of a greedy pursuit algorithm for the same goal. Key features of our Learned Greedy Method (LGM) are the ability to accommodate a dynamic number of unfolded layers.
  arXiv  Detail & Related papers  (2020-10-14T13:17:02Z)
• Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives [73.15276998621582]
  We propose a generic feature learning mechanism to advance CNN training with enhanced generalization ability. Partially inspired by DSN, we fork delicately designed side branches from the intermediate layers of a given neural network. Experiments on both category and instance recognition tasks demonstrate the substantial improvements of our proposed method.
  arXiv  Detail & Related papers  (2020-03-24T09:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.