The Curious Case of Convex Neural Networks
- URL: http://arxiv.org/abs/2006.05103v3
- Date: Sat, 10 Jul 2021 10:51:29 GMT
- Title: The Curious Case of Convex Neural Networks
- Authors: Sarath Sivaprasad, Ankur Singh, Naresh Manwani, Vineet Gandhi
- Abstract summary: We show that the convexity constraints can be enforced on both fully connected and convolutional layers.
We draw three valuable insights: (a) Input Output Convex Neural Networks (IOC-NNs) self regularize and reduce the problem of overfitting; (b) Although heavily constrained, they outperform the base multi layer perceptrons and achieve similar performance as compared to base convolutional architectures.
- Score: 12.56278477726461
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this paper, we investigate a constrained formulation of neural networks
where the output is a convex function of the input. We show that the convexity
constraints can be enforced on both fully connected and convolutional layers,
making them applicable to most architectures. The convexity constraints include
restricting the weights (for all but the first layer) to be non-negative and
using a non-decreasing convex activation function. Albeit simple, these
constraints have profound implications on the generalization abilities of the
network. We draw three valuable insights: (a) Input Output Convex Neural
Networks (IOC-NNs) self regularize and reduce the problem of overfitting; (b)
Although heavily constrained, they outperform the base multi layer perceptrons
and achieve similar performance as compared to base convolutional architectures
and (c) IOC-NNs show robustness to noise in train labels. We demonstrate the
efficacy of the proposed idea using thorough experiments and ablation studies
on standard image classification datasets with three different neural network
architectures.
Related papers
- LinSATNet: The Positive Linear Satisfiability Neural Networks [116.65291739666303]
This paper studies how to introduce the popular positive linear satisfiability to neural networks.
We propose the first differentiable satisfiability layer based on an extension of the classic Sinkhorn algorithm for jointly encoding multiple sets of marginal distributions.
arXiv Detail & Related papers (2024-07-18T22:05:21Z) - Information-Theoretic Generalization Bounds for Deep Neural Networks [22.87479366196215]
Deep neural networks (DNNs) exhibit an exceptional capacity for generalization in practical applications.
This work aims to capture the effect and benefits of depth for supervised learning via information-theoretic generalization bounds.
arXiv Detail & Related papers (2024-04-04T03:20:35Z) - LogicMP: A Neuro-symbolic Approach for Encoding First-order Logic Constraints [42.16663204729038]
This paper proposes a novel neural layer, LogicMP, whose layers perform mean-field variational inference over an MLN.
It can be plugged into any off-the-shelf neural network to encode FOLCs while retaining modularity and efficiency.
Empirical results in three kinds of tasks over graphs, images, and text show that LogicMP outperforms advanced competitors in both performance and efficiency.
arXiv Detail & Related papers (2023-09-27T07:52:30Z) - Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification.
Our results indicate that interpolating with smoother functions leads to better generalization.
arXiv Detail & Related papers (2023-05-30T19:37:44Z) - Globally Optimal Training of Neural Networks with Threshold Activation
Functions [63.03759813952481]
We study weight decay regularized training problems of deep neural networks with threshold activations.
We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
arXiv Detail & Related papers (2023-03-06T18:59:13Z) - Towards Practical Control of Singular Values of Convolutional Layers [65.25070864775793]
Convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control.
Recent research demonstrated that singular values of convolutional layers significantly affect such elusive properties.
We offer a principled approach to alleviating constraints of the prior art at the expense of an insignificant reduction in layer expressivity.
arXiv Detail & Related papers (2022-11-24T19:09:44Z) - Robust Training and Verification of Implicit Neural Networks: A
Non-Euclidean Contractive Approach [64.23331120621118]
This paper proposes a theoretical and computational framework for training and robustness verification of implicit neural networks.
We introduce a related embedded network and show that the embedded network can be used to provide an $ell_infty$-norm box over-approximation of the reachable sets of the original network.
We apply our algorithms to train implicit neural networks on the MNIST dataset and compare the robustness of our models with the models trained via existing approaches in the literature.
arXiv Detail & Related papers (2022-08-08T03:13:24Z) - CoNSoLe: Convex Neural Symbolic Learning [45.08051574243274]
Learning the underlying equation from data is a fundamental problem in many disciplines.
Recent advances rely on Neural Networks (NNs) but do not provide theoretical guarantees.
arXiv Detail & Related papers (2022-06-01T06:38:03Z) - Deep Architecture Connectivity Matters for Its Convergence: A
Fine-Grained Analysis [94.64007376939735]
We theoretically characterize the impact of connectivity patterns on the convergence of deep neural networks (DNNs) under gradient descent training.
We show that by a simple filtration on "unpromising" connectivity patterns, we can trim down the number of models to evaluate.
arXiv Detail & Related papers (2022-05-11T17:43:54Z) - Convexifying Sparse Interpolation with Infinitely Wide Neural Networks:
An Atomic Norm Approach [4.380224449592902]
This work examines the problem of exact data via sparse (neuron count), infinitely wide, single hidden layer neural networks with leaky rectified linear unit activations.
We derive simple characterizations of the convex hulls of the corresponding atomic sets for this problem under several different constraints on the weights and biases of the network.
A modest extension of our proposed framework to a binary classification problem is also presented.
arXiv Detail & Related papers (2020-07-15T21:40:51Z) - Tensor decomposition to Compress Convolutional Layers in Deep Learning [5.199454801210509]
We propose to use CP-decomposition to approximately compress the convolutional layer (CPAC-Conv layer) in deep learning.
The contributions of our work could be summarized into three aspects: (1) we adapt CP-decomposition to compress convolutional kernels and derive the expressions of both forward and backward propagations for our proposed CPAC-Conv layer; (2) compared with the original convolutional layer, the proposed CPAC-Conv layer can reduce the number of parameters without decaying prediction performance; and (3) the value of decomposed kernels indicates the significance of the corresponding feature map.
arXiv Detail & Related papers (2020-05-28T02:35:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.