Resilience of Entropy Model in Distributed Neural Networks

• URL: http://arxiv.org/abs/2403.00942v2
• Date: Thu, 11 Jul 2024 13:51:56 GMT
• Title: Resilience of Entropy Model in Distributed Neural Networks
• Abstract summary: entropy coding has been introduced to further reduce the communication overhead. In this paper, we investigate resilience of entropy models to intentional interference and unintentional interference. We propose a new defense mechanism that can reduce the transmission overhead of the attacked input by about 9% compared to unperturbed data, with only about 2% accuracy loss.
• Score: 7.6340310234573465
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distributed deep neural networks (DNNs) have emerged as a key technique to reduce communication overhead without sacrificing performance in edge computing systems. Recently, entropy coding has been introduced to further reduce the communication overhead. The key idea is to train the distributed DNN jointly with an entropy model, which is used as side information during inference time to adaptively encode latent representations into bit streams with variable length. To the best of our knowledge, the resilience of entropy models is yet to be investigated. As such, in this paper we formulate and investigate the resilience of entropy models to intentional interference (e.g., adversarial attacks) and unintentional interference (e.g., weather changes and motion blur). Through an extensive experimental campaign with 3 different DNN architectures, 2 entropy models and 4 rate-distortion trade-off factors, we demonstrate that the entropy attacks can increase the communication overhead by up to 95%. By separating compression features in frequency and spatial domain, we propose a new defense mechanism that can reduce the transmission overhead of the attacked input by about 9% compared to unperturbed data, with only about 2% accuracy loss. Importantly, the proposed defense mechanism is a standalone approach which can be applied in conjunction with approaches such as adversarial training to further improve robustness. Code will be shared for reproducibility.

Related papers

• Defending Deep Neural Networks against Backdoor Attacks via Module Switching [15.979018992591032]
  An exponential increase in the parameters of Deep Neural Networks (DNNs) has significantly raised the cost of independent training. Open-source models are more vulnerable to malicious threats, such as backdoor attacks. We propose a novel module-switching strategy to break such spurious correlations within the model's propagation path.
  arXiv  Detail & Related papers  (2025-04-08T11:01:07Z)
• Weight transport through spike timing for robust local gradients [0.5236468296934584]
  plasticity in functional neural networks is frequently expressed as gradient descent on a cost. This imposes symmetry constraints that are difficult to reconcile with local computation. We introduce spike-based alignment learning, which uses spike timing statistics to extract and correct the asymmetry between effective reciprocal connections.
  arXiv  Detail & Related papers  (2025-03-04T14:05:39Z)
• Causal Context Adjustment Loss for Learned Image Compression [72.7300229848778]
  In recent years, learned image compression (LIC) technologies have surpassed conventional methods notably in terms of rate-distortion (RD) performance. Most present techniques are VAE-based with an autoregressive entropy model, which obviously promotes the RD performance by utilizing the decoded causal context. In this paper, we make the first attempt in investigating the way to explicitly adjust the causal context with our proposed Causal Context Adjustment loss.
  arXiv  Detail & Related papers  (2024-10-07T09:08:32Z)
• Entropy Loss: An Interpretability Amplifier of 3D Object Detection Network for Intelligent Driving [12.785134710797244]
  This paper introduces a novel type of loss function, termed "Entropy Loss," along with an innovative training strategy. Entropy Loss is formulated based on the functionality of feature compression networks within the perception model. The accuracy of 3D object detection models using Entropy Loss on the KITTI test set improved by up to 4.47% compared to models without Entropy Loss.
  arXiv  Detail & Related papers  (2024-09-01T20:55:50Z)
• Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy [55.014926694758195]
  Entropy and mutual information in neural networks provide rich information on the learning process. We leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. We show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data.
  arXiv  Detail & Related papers  (2023-12-04T01:32:42Z)
• Deep Neural Networks Tend To Extrapolate Predictably [51.303814412294514]
  neural network predictions tend to be unpredictable and overconfident when faced with out-of-distribution (OOD) inputs. We observe that neural network predictions often tend towards a constant value as input data becomes increasingly OOD. We show how one can leverage our insights in practice to enable risk-sensitive decision-making in the presence of OOD inputs.
  arXiv  Detail & Related papers  (2023-10-02T03:25:32Z)
• Towards a Better Theoretical Understanding of Independent Subnetwork Training [56.24689348875711]
  We take a closer theoretical look at Independent Subnetwork Training (IST) IST is a recently proposed and highly effective technique for solving the aforementioned problems. We identify fundamental differences between IST and alternative approaches, such as distributed methods with compressed communication.
  arXiv  Detail & Related papers  (2023-06-28T18:14:22Z)
• FLARE: Detection and Mitigation of Concept Drift for Federated Learning based IoT Deployments [2.7776688429637466]
  FLARE is a lightweight dual-scheduler FL framework that conditionally transfers training data and deploys models between edge and sensor endpoints. We show that FLARE can significantly reduce the amount of data exchanged between edge and sensor nodes compared to fixed-interval scheduling methods. It can successfully detect concept drift reactively with at least a 16x reduction in latency.
  arXiv  Detail & Related papers  (2023-05-15T10:09:07Z)
• Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression [25.96187914295921]
  This paper proposes a powerful entropy model which efficiently captures both spatial and temporal dependencies. Our entropy model can achieve 18.2% saving on UVG dataset when compared with H266 (VTM) using the highest compression ratio.
  arXiv  Detail & Related papers  (2022-07-13T00:03:54Z)
• Distributed Adversarial Training to Robustify Deep Neural Networks at Scale [100.19539096465101]
  Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification. To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training. We propose a large-batch adversarial training framework implemented over multiple machines.
  arXiv  Detail & Related papers  (2022-06-13T15:39:43Z)
• Multi-fidelity Bayesian Neural Networks: Algorithms and Applications [0.0]
  We propose a new class of Bayesian neural networks (BNNs) that can be trained using noisy data of variable fidelity. We apply them to learn function approximations as well as to solve inverse problems based on partial differential equations (PDEs)
  arXiv  Detail & Related papers  (2020-12-19T02:03:53Z)
• Network Diffusions via Neural Mean-Field Dynamics [52.091487866968286]
  We propose a novel learning framework for inference and estimation problems of diffusion on networks. Our framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities. Our approach is versatile and robust to variations of the underlying diffusion network models.
  arXiv  Detail & Related papers  (2020-06-16T18:45:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.