Scaling Up Exact Neural Network Compression by ReLU Stability

• URL: http://arxiv.org/abs/2102.07804v1
• Date: Mon, 15 Feb 2021 19:19:02 GMT
• Title: Scaling Up Exact Neural Network Compression by ReLU Stability
• Abstract summary: We introduce an algorithm based on solving a single optimization problem to identify all stable neurons. Our approach is on median 21 times faster than the state-of-art method.
• Score: 22.821726425339797
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We can compress a neural network while exactly preserving its underlying functionality with respect to a given input domain if some of its neurons are stable. However, current approaches to determine the stability of neurons in networks with Rectified Linear Unit (ReLU) activations require solving or finding a good approximation to multiple discrete optimization problems. In this work, we introduce an algorithm based on solving a single optimization problem to identify all stable neurons. Our approach is on median 21 times faster than the state-of-art method, which allows us to explore exact compression on deeper (5 x 100) and wider (2 x 800) networks within minutes. For classifiers trained under an amount of L1 regularization that does not worsen accuracy, we can remove up to 40% of the connections.

Related papers

• Formulations and scalability of neural network surrogates in nonlinear optimization problems [0.0]
  We compare full-space, reduced-space, and gray-box formulations for representing trained neural networks in nonlinear constrained optimization problems. We test these formulations on a transient stability-constrained, security-constrained alternating current optimal power flow (SCOPF) problem. We solve our test problem with our largest neural network surrogate in 2.5$times$ the time required for a simpler SCOPF problem without the stability constraint.
  arXiv  Detail & Related papers  (2024-12-16T03:09:06Z)
• LinSATNet: The Positive Linear Satisfiability Neural Networks [116.65291739666303]
  This paper studies how to introduce the popular positive linear satisfiability to neural networks. We propose the first differentiable satisfiability layer based on an extension of the classic Sinkhorn algorithm for jointly encoding multiple sets of marginal distributions.
  arXiv  Detail & Related papers  (2024-07-18T22:05:21Z)
• Minimum number of neurons in fully connected layers of a given neural network (the first approximation) [0.0]
  The paper presents an algorithm for searching for the minimum number of neurons in fully connected layers of an arbitrary network solving given problem. The proposed algorithm is the first approximation for estimating the minimum number of neurons in the layer, since, on the one hand, the algorithm does not guarantee that a neural network with the found number of neurons can be trained to the required quality.
  arXiv  Detail & Related papers  (2024-05-23T03:46:07Z)
• DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization [51.517956081644186]
  We introduce a new graph-based diffusion framework, namely DIFUSCO. Our framework casts NPC problems as discrete 0, 1-vector optimization problems. For the MIS problem, DIFUSCO outperforms the previous state-of-the-art neural solver on the challenging SATLIB benchmark.
  arXiv  Detail & Related papers  (2023-02-16T11:13:36Z)
• Zonotope Domains for Lagrangian Neural Network Verification [102.13346781220383]
  We decompose the problem of verifying a deep neural network into the verification of many 2-layer neural networks. Our technique yields bounds that improve upon both linear programming and Lagrangian-based verification techniques.
  arXiv  Detail & Related papers  (2022-10-14T19:31:39Z)
• Adaptive neural domain refinement for solving time-dependent differential equations [0.0]
  A classic approach for solving differential equations with neural networks builds upon neural forms, which employ the differential equation with a discretisation of the solution domain. It would be desirable to transfer such important and successful strategies to the field of neural network based solutions. We propose a novel adaptive neural approach to meet this aim for solving time-dependent problems.
  arXiv  Detail & Related papers  (2021-12-23T13:19:07Z)
• Acceleration techniques for optimization over trained neural network ensembles [1.0323063834827415]
  We study optimization problems where the objective function is modeled through feedforward neural networks with rectified linear unit activation. We present a mixed-integer linear program based on existing popular big-$M$ formulations for optimizing over a single neural network.
  arXiv  Detail & Related papers  (2021-12-13T20:50:54Z)
• Reachability analysis of neural networks using mixed monotonicity [0.0]
  We present a new reachability analysis tool to compute an interval over-approximation of the output set of a feedforward neural network under given input uncertainty. The proposed approach adapts to neural networks an existing mixed-monotonicity method for the reachability analysis of dynamical systems.
  arXiv  Detail & Related papers  (2021-11-15T11:35:18Z)
• DeepSplit: Scalable Verification of Deep Neural Networks via Operator Splitting [70.62923754433461]
  Analyzing the worst-case performance of deep neural networks against input perturbations amounts to solving a large-scale non- optimization problem. We propose a novel method that can directly solve a convex relaxation of the problem to high accuracy, by splitting it into smaller subproblems that often have analytical solutions.
  arXiv  Detail & Related papers  (2021-06-16T20:43:49Z)
• Non-Gradient Manifold Neural Network [79.44066256794187]
  Deep neural network (DNN) generally takes thousands of iterations to optimize via gradient descent. We propose a novel manifold neural network based on non-gradient optimization.
  arXiv  Detail & Related papers  (2021-06-15T06:39:13Z)
• Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks [50.42141893913188]
  We study a distributed variable for large-scale AUC for a neural network as with a deep neural network. Our model requires a much less number of communication rounds and still a number of communication rounds in theory. Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
  arXiv  Detail & Related papers  (2020-05-05T18:08:23Z)
• Accurate Tumor Tissue Region Detection with Accelerated Deep Convolutional Neural Networks [12.7414209590152]
  Manual annotation of pathology slides for cancer diagnosis is laborious and repetitive. Our approach, (FLASH) is based on a Deep Convolutional Neural Network (DCNN) architecture. It reduces computational costs and is faster than typical deep learning approaches by two orders of magnitude.
  arXiv  Detail & Related papers  (2020-04-18T08:24:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.