Improving Convolutional Neural Networks Via Conservative Field Regularisation and Integration

• URL: http://arxiv.org/abs/2003.05182v1
• Date: Wed, 11 Mar 2020 09:29:48 GMT
• Title: Improving Convolutional Neural Networks Via Conservative Field Regularisation and Integration
• Authors: Dominique Beaini, Sofiane Achiche, Maxime Raison
• Abstract summary: Green's function (GF) is the first operation that regularizes the 2D or 3D feature space by forcing it to be conservative and physically interpretable. Our results show that such regularization allows the network to learn faster, to have smoother training curves and to better generalize, without any additional parameter.
• Score: 3.5665681694253903
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Current research in convolutional neural networks (CNN) focuses mainly on changing the architecture of the networks, optimizing the hyper-parameters and improving the gradient descent. However, most work use only 3 standard families of operations inside the CNN, the convolution, the activation function, and the pooling. In this work, we propose a new family of operations based on the Green's function of the Laplacian, which allows the network to solve the Laplacian, to integrate any vector field and to regularize the field by forcing it to be conservative. Hence, the Green's function (GF) is the first operation that regularizes the 2D or 3D feature space by forcing it to be conservative and physically interpretable, instead of regularizing the norm of the weights. Our results show that such regularization allows the network to learn faster, to have smoother training curves and to better generalize, without any additional parameter. The current manuscript presents early results, more work is required to benchmark the proposed method.

Related papers

• Kolmogorov-Arnold Transformer [72.88137795439407]
  We introduce the Kolmogorov-Arnold Transformer (KAT), a novel architecture that replaces layers with Kolmogorov-Arnold Network (KAN) layers. We identify three key challenges: (C1) Base function, (C2) Inefficiency, and (C3) Weight. With these designs, KAT outperforms traditional-based transformers.
  arXiv  Detail & Related papers  (2024-09-16T17:54:51Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Consensus Function from an $L_p^q-$norm Regularization Term for its Use as Adaptive Activation Functions in Neural Networks [0.0]
  We propose the definition and utilization of an implicit, parametric, non-linear activation function that adapts its shape during the training process. This fact increases the space of parameters to optimize within the network, but it allows a greater flexibility and generalizes the concept of neural networks. Preliminary results show that the use of these neural networks with this type of adaptive activation functions reduces the error in regression and classification examples.
  arXiv  Detail & Related papers  (2022-06-30T04:48:14Z)
• Learning Smooth Neural Functions via Lipschitz Regularization [92.42667575719048]
  We introduce a novel regularization designed to encourage smooth latent spaces in neural fields. Compared with prior Lipschitz regularized networks, ours is computationally fast and can be implemented in four lines of code.
  arXiv  Detail & Related papers  (2022-02-16T21:24:54Z)
• Graph-adaptive Rectified Linear Unit for Graph Neural Networks [64.92221119723048]
  Graph Neural Networks (GNNs) have achieved remarkable success by extending traditional convolution to learning on non-Euclidean data. We propose Graph-adaptive Rectified Linear Unit (GReLU) which is a new parametric activation function incorporating the neighborhood information in a novel and efficient way. We conduct comprehensive experiments to show that our plug-and-play GReLU method is efficient and effective given different GNN backbones and various downstream tasks.
  arXiv  Detail & Related papers  (2022-02-13T10:54:59Z)
• Power Flow Balancing with Decentralized Graph Neural Networks [4.812718493682454]
  We propose an end-to-end framework based on a Graph Neural Network (GNN) to balance the power flows in a generic grid. The proposed framework is efficient and, compared to other solvers based on deep learning, is robust to perturbations not only to the physical quantities on the grid components, but also to the topology.
  arXiv  Detail & Related papers  (2021-11-03T12:14:56Z)
• Orthogonal Graph Neural Networks [53.466187667936026]
  Graph neural networks (GNNs) have received tremendous attention due to their superiority in learning node representations. stacking more convolutional layers significantly decreases the performance of GNNs. We propose a novel Ortho-GConv, which could generally augment the existing GNN backbones to stabilize the model training and improve the model's generalization performance.
  arXiv  Detail & Related papers  (2021-09-23T12:39:01Z)
• Gradient Centralization: A New Optimization Technique for Deep Neural Networks [74.935141515523]
  gradient centralization (GC) operates directly on gradients by centralizing the gradient vectors to have zero mean. GC can be viewed as a projected gradient descent method with a constrained loss function. GC is very simple to implement and can be easily embedded into existing gradient based DNNs with only one line of code.
  arXiv  Detail & Related papers  (2020-04-03T10:25:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.