Learning DNN networks using un-rectifying ReLU with compressed sensing application

• URL: http://arxiv.org/abs/2101.06940v1
• Date: Mon, 18 Jan 2021 09:04:37 GMT
• Title: Learning DNN networks using un-rectifying ReLU with compressed sensing application
• Authors: Wen-Liang Hwang, Shih-Shuo Tung
• Abstract summary: The ReLU network in this study was un-rectified. In experiments, our novel approach to solving the compressed sensing recovery problem achieved state-of-the-art performance.
• Score: 4.111899441919165
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The un-rectifying technique expresses a non-linear point-wise activation function as a data-dependent variable, which means that the activation variable along with its input and output can all be employed in optimization. The ReLU network in this study was un-rectified means that the activation functions could be replaced with data-dependent activation variables in the form of equations and constraints. The discrete nature of activation variables associated with un-rectifying ReLUs allows the reformulation of deep learning problems as problems of combinatorial optimization. However, we demonstrate that the optimal solution to a combinatorial optimization problem can be preserved by relaxing the discrete domains of activation variables to closed intervals. This makes it easier to learn a network using methods developed for real-domain constrained optimization. We also demonstrate that by introducing data-dependent slack variables as constraints, it is possible to optimize a network based on the augmented Lagrangian approach. This means that our method could theoretically achieve global convergence and all limit points are critical points of the learning problem. In experiments, our novel approach to solving the compressed sensing recovery problem achieved state-of-the-art performance when applied to the MNIST database and natural images.

Related papers

• Learning to Optimize with Stochastic Dominance Constraints [103.26714928625582]
  In this paper, we develop a simple yet efficient approach for the problem of comparing uncertain quantities. We recast inner optimization in the Lagrangian as a learning problem for surrogate approximation, which bypasses apparent intractability. The proposed light-SD demonstrates superior performance on several representative problems ranging from finance to supply chain management.
  arXiv  Detail & Related papers  (2022-11-14T21:54:31Z)
• Stabilizing Q-learning with Linear Architectures for Provably Efficient Learning [53.17258888552998]
  This work proposes an exploration variant of the basic $Q$-learning protocol with linear function approximation. We show that the performance of the algorithm degrades very gracefully under a novel and more permissive notion of approximation error.
  arXiv  Detail & Related papers  (2022-06-01T23:26:51Z)
• Level set learning with pseudo-reversible neural networks for nonlinear dimension reduction in function approximation [8.28646586439284]
  We propose a new method of Dimension Reduction via Learning Level Sets (DRiLLS) for function approximation. Our method contains two major components: one is the pseudo-reversible neural network (PRNN) module that effectively transforms high-dimensional input variables to low-dimensional active variables. The PRNN not only relaxes the invertibility constraint of the nonlinear transformation present in the NLL method due to the use of RevNet, but also adaptively weights the influence of each sample and controls the sensitivity the function to the learned active variables.
  arXiv  Detail & Related papers  (2021-12-02T17:25:34Z)
• Physics and Equality Constrained Artificial Neural Networks: Application to Partial Differential Equations [1.370633147306388]
  Physics-informed neural networks (PINNs) have been proposed to learn the solution of partial differential equations (PDE) Here, we show that this specific way of formulating the objective function is the source of severe limitations in the PINN approach. We propose a versatile framework that can tackle both inverse and forward problems.
  arXiv  Detail & Related papers  (2021-09-30T05:55:35Z)
• Differentiable Causal Discovery from Interventional Data [141.41931444927184]
  We propose a theoretically-grounded method based on neural networks that can leverage interventional data. We show that our approach compares favorably to the state of the art in a variety of settings.
  arXiv  Detail & Related papers  (2020-07-03T15:19:17Z)
• Constrained Combinatorial Optimization with Reinforcement Learning [0.30938904602244344]
  This paper presents a framework to tackle constrained optimization problems using deep Reinforcement Learning (RL) We extend the Neural Combinatorial Optimization (NCO) theory in order to deal with constraints in its formulation. In that context, the solution is iteratively constructed based on interactions with the environment.
  arXiv  Detail & Related papers  (2020-06-22T03:13:07Z)
• Cogradient Descent for Bilinear Optimization [124.45816011848096]
  We introduce a Cogradient Descent algorithm (CoGD) to address the bilinear problem. We solve one variable by considering its coupling relationship with the other, leading to a synchronous gradient descent. Our algorithm is applied to solve problems with one variable under the sparsity constraint.
  arXiv  Detail & Related papers  (2020-06-16T13:41:54Z)
• Joint learning of variational representations and solvers for inverse problems with partially-observed data [13.984814587222811]
  In this paper, we design an end-to-end framework allowing to learn actual variational frameworks for inverse problems in a supervised setting. The variational cost and the gradient-based solver are both stated as neural networks using automatic differentiation for the latter. This leads to a data-driven discovery of variational models.
  arXiv  Detail & Related papers  (2020-06-05T19:53:34Z)
• Neural Control Variates [71.42768823631918]
  We show that a set of neural networks can face the challenge of finding a good approximation of the integrand. We derive a theoretically optimal, variance-minimizing loss function, and propose an alternative, composite loss for stable online training in practice. Specifically, we show that the learned light-field approximation is of sufficient quality for high-order bounces, allowing us to omit the error correction and thereby dramatically reduce the noise at the cost of negligible visible bias.
  arXiv  Detail & Related papers  (2020-06-02T11:17:55Z)
• Optimizing Wireless Systems Using Unsupervised and Reinforced-Unsupervised Deep Learning [96.01176486957226]
  Resource allocation and transceivers in wireless networks are usually designed by solving optimization problems. In this article, we introduce unsupervised and reinforced-unsupervised learning frameworks for solving both variable and functional optimization problems.
  arXiv  Detail & Related papers  (2020-01-03T11:01:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.