Parallelization Techniques for Verifying Neural Networks
- URL: http://arxiv.org/abs/2004.08440v3
- Date: Fri, 21 Aug 2020 16:15:13 GMT
- Title: Parallelization Techniques for Verifying Neural Networks
- Authors: Haoze Wu, Alex Ozdemir, Aleksandar Zelji\'c, Ahmed Irfan, Kyle Julian,
Divya Gopinath, Sadjad Fouladi, Guy Katz, Corina Pasareanu and Clark Barrett
- Abstract summary: We introduce an algorithm based on the verification problem in an iterative manner and explore two partitioning strategies.
We also introduce a highly parallelizable pre-processing algorithm that uses the neuron activation phases to simplify the neural network verification problems.
- Score: 52.917845265248744
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Inspired by recent successes with parallel optimization techniques for
solving Boolean satisfiability, we investigate a set of strategies and
heuristics that aim to leverage parallel computing to improve the scalability
of neural network verification. We introduce an algorithm based on partitioning
the verification problem in an iterative manner and explore two partitioning
strategies, that work by partitioning the input space or by case splitting on
the phases of the neuron activations, respectively. We also introduce a highly
parallelizable pre-processing algorithm that uses the neuron activation phases
to simplify the neural network verification problems. An extensive experimental
evaluation shows the benefit of these techniques on both existing benchmarks
and new benchmarks from the aviation domain. A preliminary experiment with
ultra-scaling our algorithm using a large distributed cloud-based platform also
shows promising results.
Related papers
- [Experiments & Analysis] Evaluating the Feasibility of Sampling-Based Techniques for Training Multilayer Perceptrons [10.145355763143218]
Several sampling-based techniques have been proposed for speeding up the training time of deep neural networks.
These techniques fall under two categories: (i) sampling a subset of nodes in every hidden layer as active at every iteration and (ii) sampling a subset of nodes from the previous layer to approximate the current layer's activations.
In this paper, we evaluate the feasibility of these approaches on CPU machines with limited computational resources.
arXiv Detail & Related papers (2023-06-15T17:19:48Z) - Stochastic Unrolled Federated Learning [85.6993263983062]
We introduce UnRolled Federated learning (SURF), a method that expands algorithm unrolling to federated learning.
Our proposed method tackles two challenges of this expansion, namely the need to feed whole datasets to the unrolleds and the decentralized nature of federated learning.
arXiv Detail & Related papers (2023-05-24T17:26:22Z) - The Cascaded Forward Algorithm for Neural Network Training [61.06444586991505]
We propose a new learning framework for neural networks, namely Cascaded Forward (CaFo) algorithm, which does not rely on BP optimization as that in FF.
Unlike FF, our framework directly outputs label distributions at each cascaded block, which does not require generation of additional negative samples.
In our framework each block can be trained independently, so it can be easily deployed into parallel acceleration systems.
arXiv Detail & Related papers (2023-03-17T02:01:11Z) - Globally Optimal Training of Neural Networks with Threshold Activation
Functions [63.03759813952481]
We study weight decay regularized training problems of deep neural networks with threshold activations.
We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
arXiv Detail & Related papers (2023-03-06T18:59:13Z) - On the Convergence of Distributed Stochastic Bilevel Optimization
Algorithms over a Network [55.56019538079826]
Bilevel optimization has been applied to a wide variety of machine learning models.
Most existing algorithms restrict their single-machine setting so that they are incapable of handling distributed data.
We develop novel decentralized bilevel optimization algorithms based on a gradient tracking communication mechanism and two different gradients.
arXiv Detail & Related papers (2022-06-30T05:29:52Z) - Acceleration techniques for optimization over trained neural network
ensembles [1.0323063834827415]
We study optimization problems where the objective function is modeled through feedforward neural networks with rectified linear unit activation.
We present a mixed-integer linear program based on existing popular big-$M$ formulations for optimizing over a single neural network.
arXiv Detail & Related papers (2021-12-13T20:50:54Z) - Ada-SISE: Adaptive Semantic Input Sampling for Efficient Explanation of
Convolutional Neural Networks [26.434705114982584]
We propose an efficient interpretation method for convolutional neural networks.
Experimental results show that the proposed method can reduce the execution time up to 30%.
arXiv Detail & Related papers (2021-02-15T19:10:00Z) - Topological Gradient-based Competitive Learning [1.6752712949948443]
This work is to present a novel comprehensive theory aspiring at bridging competitive learning with gradient-based learning.
We fully demonstrate the theoretical equivalence of two novel gradient-based competitive layers.
Preliminary experiments show how the dual approach, trained on the transpose of the input matrix, lead to faster convergence rate and higher training accuracy both in low and high-dimensional scenarios.
arXiv Detail & Related papers (2020-08-21T13:44:38Z) - Binary Neural Networks: A Survey [126.67799882857656]
The binary neural network serves as a promising technique for deploying deep models on resource-limited devices.
The binarization inevitably causes severe information loss, and even worse, its discontinuity brings difficulty to the optimization of the deep network.
We present a survey of these algorithms, mainly categorized into the native solutions directly conducting binarization, and the optimized ones using techniques like minimizing the quantization error, improving the network loss function, and reducing the gradient error.
arXiv Detail & Related papers (2020-03-31T16:47:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.