Convex Regularization Behind Neural Reconstruction

• URL: http://arxiv.org/abs/2012.05169v1
• Date: Wed, 9 Dec 2020 16:57:16 GMT
• Title: Convex Regularization Behind Neural Reconstruction
• Authors: Arda Sahiner, Morteza Mardani, Batu Ozturkler, Mert Pilanci, John Pauly
• Abstract summary: This paper advocates a convex duality framework to make neural networks amenable to convex solvers. Experiments with MNIST fastMRI datasets confirm the efficacy of the dual network optimization problem.
• Score: 21.369208659395042
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural networks have shown tremendous potential for reconstructing high-resolution images in inverse problems. The non-convex and opaque nature of neural networks, however, hinders their utility in sensitive applications such as medical imaging. To cope with this challenge, this paper advocates a convex duality framework that makes a two-layer fully-convolutional ReLU denoising network amenable to convex optimization. The convex dual network not only offers the optimum training with convex solvers, but also facilitates interpreting training and prediction. In particular, it implies training neural networks with weight decay regularization induces path sparsity while the prediction is piecewise linear filtering. A range of experiments with MNIST and fastMRI datasets confirm the efficacy of the dual network optimization problem.

Related papers

• Convex Formulations for Training Two-Layer ReLU Neural Networks [21.88871868680998]
  Non-layer, NP-hard optimization problems are crucial for machine learning models. We introduce a semidefinite relaxation that can be solved in a finite-width two neural network.
  arXiv  Detail & Related papers  (2024-10-29T17:53:15Z)
• Improving Generalization of Deep Neural Networks by Optimum Shifting [33.092571599896814]
  We propose a novel method called emphoptimum shifting, which changes the parameters of a neural network from a sharp minimum to a flatter one. Our method is based on the observation that when the input and output of a neural network are fixed, the matrix multiplications within the network can be treated as systems of under-determined linear equations.
  arXiv  Detail & Related papers  (2024-05-23T02:31:55Z)
• Fixing the NTK: From Neural Network Linearizations to Exact Convex Programs [63.768739279562105]
  We show that for a particular choice of mask weights that do not depend on the learning targets, this kernel is equivalent to the NTK of the gated ReLU network on the training data. A consequence of this lack of dependence on the targets is that the NTK cannot perform better than the optimal MKL kernel on the training set.
  arXiv  Detail & Related papers  (2023-09-26T17:42:52Z)
• Convex Dual Theory Analysis of Two-Layer Convolutional Neural Networks with Soft-Thresholding [15.514556290714053]
  Soft-thresholding has been widely used in neural networks. A new way to convexify soft neuraling networks is presented. This work provides a new way to convexify soft neuraling networks.
  arXiv  Detail & Related papers  (2023-04-14T07:04:07Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Training Adaptive Reconstruction Networks for Blind Inverse Problems [0.0]
  We show through various applications that training the network with a family of forward operators allows solving the adaptivity problem without compromising the reconstruction quality significantly. Experiments include partial Fourier sampling problems arising in magnetic resonance imaging (MRI) with sensitivity estimation and off-resonance effects, computerized tomography (CT) with a tilted geometry and image deblurring with Fresnel diffraction kernels.
  arXiv  Detail & Related papers  (2022-02-23T07:56:02Z)
• Optimization-Based Separations for Neural Networks [57.875347246373956]
  We show that gradient descent can efficiently learn ball indicator functions using a depth 2 neural network with two layers of sigmoidal activations. This is the first optimization-based separation result where the approximation benefits of the stronger architecture provably manifest in practice.
  arXiv  Detail & Related papers  (2021-12-04T18:07:47Z)
• LocalDrop: A Hybrid Regularization for Deep Neural Networks [98.30782118441158]
  We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs) has been developed based on the proposed upper bound of the local Rademacher complexity.
  arXiv  Detail & Related papers  (2021-03-01T03:10:11Z)
• Operational vs Convolutional Neural Networks for Image Denoising [25.838282412957675]
  Convolutional Neural Networks (CNNs) have recently become a favored technique for image denoising due to its adaptive learning ability. We propose a heterogeneous network model which allows greater flexibility for embedding additional non-linearity at the core of the data transformation. An extensive set of comparative evaluations of ONNs and CNNs over two severe image denoising problems yield conclusive evidence that ONNs enriched by non-linear operators can achieve a superior denoising performance against CNNs with both equivalent and well-known deep configurations.
  arXiv  Detail & Related papers  (2020-09-01T12:15:28Z)
• The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutions [51.60996023961886]
  We prove that finding all globally optimal two-layer ReLU neural networks can be performed by solving a convex optimization program with cone constraints. Our analysis is novel, characterizes all optimal solutions, and does not leverage duality-based analysis which was recently used to lift neural network training into convex spaces.
  arXiv  Detail & Related papers  (2020-06-10T15:38:30Z)
• Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks [107.77595511218429]
  In this paper, we investigate the empirical Rademacher complexity related to intermediate layers of deep neural networks. We propose a feature distortion method (Disout) for addressing the aforementioned problem. The superiority of the proposed feature map distortion for producing deep neural network with higher testing performance is analyzed and demonstrated.
  arXiv  Detail & Related papers  (2020-02-23T13:59:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.