ResiliNet: Failure-Resilient Inference in Distributed Neural Networks
- URL: http://arxiv.org/abs/2002.07386v4
- Date: Sat, 19 Dec 2020 08:08:17 GMT
- Title: ResiliNet: Failure-Resilient Inference in Distributed Neural Networks
- Authors: Ashkan Yousefpour, Brian Q. Nguyen, Siddartha Devic, Guanhua Wang,
Aboudy Kreidieh, Hans Lobel, Alexandre M. Bayen, Jason P. Jue
- Abstract summary: We introduce ResiliNet, a scheme for making inference in distributed neural networks resilient to physical node failures.
Failout simulates physical node failure conditions during training using dropout, and is specifically designed to improve the resiliency of distributed neural networks.
- Score: 56.255913459850674
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Federated Learning aims to train distributed deep models without sharing the
raw data with the centralized server. Similarly, in distributed inference of
neural networks, by partitioning the network and distributing it across several
physical nodes, activations and gradients are exchanged between physical nodes,
rather than raw data. Nevertheless, when a neural network is partitioned and
distributed among physical nodes, failure of physical nodes causes the failure
of the neural units that are placed on those nodes, which results in a
significant performance drop. Current approaches focus on resiliency of
training in distributed neural networks. However, resiliency of inference in
distributed neural networks is less explored. We introduce ResiliNet, a scheme
for making inference in distributed neural networks resilient to physical node
failures. ResiliNet combines two concepts to provide resiliency: skip
hyperconnection, a concept for skipping nodes in distributed neural networks
similar to skip connection in resnets, and a novel technique called failout,
which is introduced in this paper. Failout simulates physical node failure
conditions during training using dropout, and is specifically designed to
improve the resiliency of distributed neural networks. The results of the
experiments and ablation studies using three datasets confirm the ability of
ResiliNet to provide inference resiliency for distributed neural networks.
Related papers
- Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
We propose to represent neural networks as computational graphs of parameters.
Our approach enables a single model to encode neural computational graphs with diverse architectures.
We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
arXiv Detail & Related papers (2024-03-18T18:01:01Z) - Seeking Next Layer Neurons' Attention for Error-Backpropagation-Like
Training in a Multi-Agent Network Framework [6.446189857311325]
We propose a local objective for neurons that align them to exhibit similarities to error-backpropagation.
We examine a neural network comprising decentralized, self-interested neurons seeking to maximize their local objective.
We demonstrate the learning capacity of these multi-agent neural networks through experiments on three datasets.
arXiv Detail & Related papers (2023-10-15T21:07:09Z) - Addressing caveats of neural persistence with deep graph persistence [54.424983583720675]
We find that the variance of network weights and spatial concentration of large weights are the main factors that impact neural persistence.
We propose an extension of the filtration underlying neural persistence to the whole neural network instead of single layers.
This yields our deep graph persistence measure, which implicitly incorporates persistent paths through the network and alleviates variance-related issues.
arXiv Detail & Related papers (2023-07-20T13:34:11Z) - Guaranteed Quantization Error Computation for Neural Network Model
Compression [2.610470075814367]
Neural network model compression techniques can address the computation issue of deep neural networks on embedded devices in industrial systems.
A merged neural network is built from a feedforward neural network and its quantized version to produce the exact output difference between two neural networks.
arXiv Detail & Related papers (2023-04-26T20:21:54Z) - Global quantitative robustness of regression feed-forward neural
networks [0.0]
We adapt the notion of the regression breakdown point to regression neural networks.
We compare the performance, measured by the out-of-sample loss, by a proxy of the breakdown rate.
The results indeed motivate to use robust loss functions for neural network training.
arXiv Detail & Related papers (2022-11-18T09:57:53Z) - Spiking neural network for nonlinear regression [68.8204255655161]
Spiking neural networks carry the potential for a massive reduction in memory and energy consumption.
They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware.
A framework for regression using spiking neural networks is proposed.
arXiv Detail & Related papers (2022-10-06T13:04:45Z) - Consistency of Neural Networks with Regularization [0.0]
This paper proposes the general framework of neural networks with regularization and prove its consistency.
Two types of activation functions: hyperbolic function(Tanh) and rectified linear unit(ReLU) have been taken into consideration.
arXiv Detail & Related papers (2022-06-22T23:33:39Z) - Deep Kronecker neural networks: A general framework for neural networks
with adaptive activation functions [4.932130498861987]
We propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions.
Under suitable conditions, KNNs induce a faster decay of the loss than that by the feed-forward networks.
arXiv Detail & Related papers (2021-05-20T04:54:57Z) - And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks.
We define AND-like neurons and propose measures to increase their proportion in the network.
Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
arXiv Detail & Related papers (2021-02-15T08:19:05Z) - Provably Training Neural Network Classifiers under Fairness Constraints [70.64045590577318]
We show that overparametrized neural networks could meet the constraints.
Key ingredient of building a fair neural network classifier is establishing no-regret analysis for neural networks.
arXiv Detail & Related papers (2020-12-30T18:46:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.