Diffusion Mechanism in Residual Neural Network: Theory and Applications

• URL: http://arxiv.org/abs/2105.03155v5
• Date: Sat, 29 Apr 2023 03:37:28 GMT
• Title: Diffusion Mechanism in Residual Neural Network: Theory and Applications
• Authors: Tangjun Wang, Zehao Dou, Chenglong Bao, Zuoqiang Shi
• Abstract summary: In many learning tasks with limited training samples, the diffusion connects the labeled and unlabeled data points. We propose a novel diffusion residual network (Diff-ResNet) internally introduces diffusion into the architectures of neural networks. Under the structured data assumption, it is proved that the proposed diffusion block can increase the distance-diameter ratio that improves the separability of inter-class points.
• Score: 12.573746641284849
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Diffusion, a fundamental internal mechanism emerging in many physical processes, describes the interaction among different objects. In many learning tasks with limited training samples, the diffusion connects the labeled and unlabeled data points and is a critical component for achieving high classification accuracy. Many existing deep learning approaches directly impose the fusion loss when training neural networks. In this work, inspired by the convection-diffusion ordinary differential equations (ODEs), we propose a novel diffusion residual network (Diff-ResNet), internally introduces diffusion into the architectures of neural networks. Under the structured data assumption, it is proved that the proposed diffusion block can increase the distance-diameter ratio that improves the separability of inter-class points and reduces the distance among local intra-class points. Moreover, this property can be easily adopted by the residual networks for constructing the separable hyperplanes. Extensive experiments of synthetic binary classification, semi-supervised graph node classification and few-shot image classification in various datasets validate the effectiveness of the proposed method.

Related papers

• Neural Message Passing Induced by Energy-Constrained Diffusion [79.9193447649011]
  We propose an energy-constrained diffusion model as a principled interpretable framework for understanding the mechanism of MPNNs. We show that the new model can yield promising performance for cases where the data structures are observed (as a graph), partially observed or completely unobserved.
  arXiv  Detail & Related papers  (2024-09-13T17:54:41Z)
• Hidden Classification Layers: Enhancing linear separability between classes in neural networks layers [0.0]
  We investigate the impact on deep network performances of a training approach. We propose a neural network architecture which induces an error function involving the outputs of all the network layers.
  arXiv  Detail & Related papers  (2023-06-09T10:52:49Z)
• Kernel function impact on convolutional neural networks [10.98068123467568]
  We study the usage of kernel functions at the different layers in a convolutional neural network. We show how one can effectively leverage kernel functions, by introducing a more distortion aware pooling layers. We propose Kernelized Dense Layers (KDL), which replace fully-connected layers.
  arXiv  Detail & Related papers  (2023-02-20T19:57:01Z)
• Discretization Invariant Networks for Learning Maps between Neural Fields [3.09125960098955]
  We present a new framework for understanding and designing discretization invariant neural networks (DI-Nets) Our analysis establishes upper bounds on the deviation in model outputs under different finite discretizations. We prove by construction that DI-Nets universally approximate a large class of maps between integrable function spaces.
  arXiv  Detail & Related papers  (2022-06-02T17:44:03Z)
• Optimization-Based Separations for Neural Networks [57.875347246373956]
  We show that gradient descent can efficiently learn ball indicator functions using a depth 2 neural network with two layers of sigmoidal activations. This is the first optimization-based separation result where the approximation benefits of the stronger architecture provably manifest in practice.
  arXiv  Detail & Related papers  (2021-12-04T18:07:47Z)
• Understanding the Distributions of Aggregation Layers in Deep Neural Networks [8.784438985280092]
  aggregation functions as an important mechanism for consolidating deep features into a more compact representation. In particular, the proximity of global aggregation layers to the output layers of DNNs mean that aggregated features have a direct influence on the performance of a deep net. We propose a novel mathematical formulation for analytically modelling the probability distributions of output values of layers involved with deep feature aggregation.
  arXiv  Detail & Related papers  (2021-07-09T14:23:57Z)
• Influence Estimation and Maximization via Neural Mean-Field Dynamics [60.91291234832546]
  We propose a novel learning framework using neural mean-field (NMF) dynamics for inference and estimation problems. Our framework can simultaneously learn the structure of the diffusion network and the evolution of node infection probabilities.
  arXiv  Detail & Related papers  (2021-06-03T00:02:05Z)
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning [50.24174211654775]
  We present a novel contrastive self-supervised learning framework for anomaly detection on attributed networks. Our framework fully exploits the local information from network data by sampling a novel type of contrastive instance pair. A graph neural network-based contrastive learning model is proposed to learn informative embedding from high-dimensional attributes and local structure.
  arXiv  Detail & Related papers  (2021-02-27T03:17:20Z)
• Network Diffusions via Neural Mean-Field Dynamics [52.091487866968286]
  We propose a novel learning framework for inference and estimation problems of diffusion on networks. Our framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities. Our approach is versatile and robust to variations of the underlying diffusion network models.
  arXiv  Detail & Related papers  (2020-06-16T18:45:20Z)
• Embedding Propagation: Smoother Manifold for Few-Shot Classification [131.81692677836202]
  We propose to use embedding propagation as an unsupervised non-parametric regularizer for manifold smoothing in few-shot classification. We empirically show that embedding propagation yields a smoother embedding manifold. We show that embedding propagation consistently improves the accuracy of the models in multiple semi-supervised learning scenarios by up to 16% points.
  arXiv  Detail & Related papers  (2020-03-09T13:51:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.