Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey

• URL: http://arxiv.org/abs/2405.01725v1
• Date: Thu, 2 May 2024 20:43:58 GMT
• Title: Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey
• Authors: Guoping Xu, Xiaxia Wang, Xinglong Wu, Xuesong Leng, Yongchao Xu,
• Abstract summary: skip connection has played an essential role in the architecture of deep neural networks. This survey provides a comprehensive summary and outlook on the development of skip connections in deep neural networks. We summarize seminal papers, source code, models, and datasets that utilize skip connections in computer vision.
• Score: 6.2299272077865675
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning has made significant progress in computer vision, specifically in image classification, object detection, and semantic segmentation. The skip connection has played an essential role in the architecture of deep neural networks,enabling easier optimization through residual learning during the training stage and improving accuracy during testing. Many neural networks have inherited the idea of residual learning with skip connections for various tasks, and it has been the standard choice for designing neural networks. This survey provides a comprehensive summary and outlook on the development of skip connections in deep neural networks. The short history of skip connections is outlined, and the development of residual learning in deep neural networks is surveyed. The effectiveness of skip connections in the training and testing stages is summarized, and future directions for using skip connections in residual learning are discussed. Finally, we summarize seminal papers, source code, models, and datasets that utilize skip connections in computer vision, including image classification, object detection, semantic segmentation, and image reconstruction. We hope this survey could inspire peer researchers in the community to develop further skip connections in various forms and tasks and the theory of residual learning in deep neural networks. The project page can be found at https://github.com/apple1986/Residual_Learning_For_Images

Related papers

• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks [74.3099028063756]
  We develop a new method with neuronal operations based on lateral connections and Hebbian learning. We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities. Our method consistently solves for spiking neural networks with nearly zero forgetting.
  arXiv  Detail & Related papers  (2024-02-19T09:29:37Z)
• Understanding Vector-Valued Neural Networks and Their Relationship with Real and Hypercomplex-Valued Neural Networks [0.4895118383237099]
  This paper aims to present a broad framework for vector-valued neural networks, referred to as V-nets. We show how V-nets, including hypercomplex-valued neural networks, can be implemented in current deep-learning libraries as real-valued networks.
  arXiv  Detail & Related papers  (2023-09-14T13:48:16Z)
• Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation? [41.58529335439799]
  The backpropagation of error algorithm used to train deep neural networks has been fundamental to the successes of deep learning. Recent work has developed the idea into a general-purpose algorithm able to train neural networks using only local computations. We show the substantially greater flexibility of predictive coding networks against equivalent deep neural networks.
  arXiv  Detail & Related papers  (2022-02-18T22:57:03Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• A Few-shot Learning Graph Multi-Trajectory Evolution Network for Forecasting Multimodal Baby Connectivity Development from a Baseline Timepoint [53.73316520733503]
  We propose a Graph Multi-Trajectory Evolution Network (GmTE-Net), which adopts a teacher-student paradigm. This is the first teacher-student architecture tailored for brain graph multi-trajectory growth prediction.
  arXiv  Detail & Related papers  (2021-10-06T08:26:57Z)
• Training Spiking Neural Networks Using Lessons From Deep Learning [28.827506468167652]
  The inner workings of our synapses and neurons provide a glimpse at what the future of deep learning might look like. Some ideas are well accepted and commonly used amongst the neuromorphic engineering community, while others are presented or justified for the first time here. A series of companion interactive tutorials complementary to this paper using our Python package, snnTorch, are also made available.
  arXiv  Detail & Related papers  (2021-09-27T09:28:04Z)
• Deep Spiking Convolutional Neural Network for Single Object Localization Based On Deep Continuous Local Learning [0.0]
  We propose a deep convolutional spiking neural network for the localization of a single object in a grayscale image. Results reported on Oxford-IIIT-Pet validates the exploitation of spiking neural networks with a supervised learning approach.
  arXiv  Detail & Related papers  (2021-05-12T12:02:05Z)
• Learning Contact Dynamics using Physically Structured Neural Networks [81.73947303886753]
  We use connections between deep neural networks and differential equations to design a family of deep network architectures for representing contact dynamics between objects. We show that these networks can learn discontinuous contact events in a data-efficient manner from noisy observations. Our results indicate that an idealised form of touch feedback is a key component of making this learning problem tractable.
  arXiv  Detail & Related papers  (2021-02-22T17:33:51Z)
• Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
  We propose a topological perspective to represent a network into a complete graph for analysis. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
  arXiv  Detail & Related papers  (2020-08-19T04:53:31Z)
• Neural Rule Ensembles: Encoding Sparse Feature Interactions into Neural Networks [3.7277730514654555]
  We use decision trees to capture relevant features and their interactions and define a mapping to encode extracted relationships into a neural network. At the same time through feature selection it enables learning of compact representations compared to state of the art tree-based approaches.
  arXiv  Detail & Related papers  (2020-02-11T11:22:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.