Assessing the Adversarial Robustness of Monte Carlo and Distillation Methods for Deep Bayesian Neural Network Classification

• URL: http://arxiv.org/abs/2002.02842v1
• Date: Fri, 7 Feb 2020 15:29:22 GMT
• Title: Assessing the Adversarial Robustness of Monte Carlo and Distillation Methods for Deep Bayesian Neural Network Classification
• Abstract summary: We show that full MCMC-based inference has excellent robustness, significantly outperforming standard point estimation-based learning. We present a storage-efficient approach to computing adversarial examples for large Monte Carlo ensembles.
• Score: 18.90540785793023
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we consider the problem of assessing the adversarial robustness of deep neural network models under both Markov chain Monte Carlo (MCMC) and Bayesian Dark Knowledge (BDK) inference approximations. We characterize the robustness of each method to two types of adversarial attacks: the fast gradient sign method (FGSM) and projected gradient descent (PGD). We show that full MCMC-based inference has excellent robustness, significantly outperforming standard point estimation-based learning. On the other hand, BDK provides marginal improvements. As an additional contribution, we present a storage-efficient approach to computing adversarial examples for large Monte Carlo ensembles using both the FGSM and PGD attacks.

Related papers

• Monte Carlo Stochastic Depth for Uncertainty Estimation in Deep Learning [0.6117371161379207]
  Deep neural networks in safety-critical systems require reliable and efficient uncertainty quantification (UQ)<n>A practical strategy for UQ is repurposing regularizers as scalable approximate Bayesian inference methods, such as Monte Carlo Dropout (MCD) and MC-DropBlock (MCDB)<n>We establish MCSD as a theoretically-grounded and empirically-validated tool for efficient Bayesian approximation in modern deep learning.
  arXiv  Detail & Related papers  (2026-04-14T13:33:40Z)
• Learning Discrete Bayesian Networks with Hierarchical Dirichlet Shrinkage [52.914168158222765]
  We detail a comprehensive Bayesian framework for learning DBNs.<n>We give a novel Markov chain Monte Carlo (MCMC) algorithm utilizing parallel Langevin proposals to generate exact posterior samples.<n>We apply our methodology to uncover prognostic network structure from primary breast cancer samples.
  arXiv  Detail & Related papers  (2025-09-16T17:24:35Z)
• WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training [64.0932926819307]
  We present Warmup-Stable and Merge (WSM), a framework that establishes a formal connection between learning rate decay and model merging.<n>WSM provides a unified theoretical foundation for emulating various decay strategies.<n>Our framework consistently outperforms the widely-adopted Warmup-Stable-Decay (WSD) approach across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-07-23T16:02:06Z)
• Humble your Overconfident Networks: Unlearning Overfitting via Sequential Monte Carlo Tempered Deep Ensembles [3.2254941904559917]
  We introduce a scalable variant by incorporating Gradient Hamiltonian Monte Carlo proposals into Sequential Monte Carlo (SMC) methods.<n>Our resulting SMCSGHMC algorithm outperforms gradient descent ensembles across image classification, out-of-distribution detection, and transfer learning tasks.
  arXiv  Detail & Related papers  (2025-05-16T20:10:04Z)
• Channel Fingerprint Construction for Massive MIMO: A Deep Conditional Generative Approach [65.47969413708344]
  We introduce the concept of CF twins and design a conditional generative diffusion model (CGDM)<n>We employ a variational inference technique to derive the evidence lower bound (ELBO) for the log-marginal distribution of the observed fine-grained CF conditioned on the coarse-grained CF.<n>We show that the proposed approach exhibits significant improvement in reconstruction performance compared to the baselines.
  arXiv  Detail & Related papers  (2025-05-12T01:36:06Z)
• Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo [32.46884330460211]
  We propose a simple yet effective approach to enhance sample diversity in Gradient Markov Chain Monte Carlo. This approach produces a more diverse set of samples, allowing faster mixing within the same computational budget. Our experiments on image classification tasks, including OOD robustness, diversity, loss surface analyses, and a comparative study with Hamiltonian Monte Carlo, demonstrate the superiority of the proposed approach.
  arXiv  Detail & Related papers  (2025-03-02T02:42:50Z)
• Uncertainty quantification for Markov chains with application to temporal difference learning [63.49764856675643]
  We develop novel high-dimensional concentration inequalities and Berry-Esseen bounds for vector- and matrix-valued functions of Markov chains. We analyze the TD learning algorithm, a widely used method for policy evaluation in reinforcement learning.
  arXiv  Detail & Related papers  (2025-02-19T15:33:55Z)
• Towards Adversarially Robust Deep Metric Learning [0.8702432681310401]
  Deep neural networks are prone to adversarial attacks and could be easily fooled by adversarial examples. Existing works fail to thoroughly inspect the robustness of DML models. We propose a new defense, the Ensemble Adversarial Training (EAT), which exploits ensemble learning and adversarial training.
  arXiv  Detail & Related papers  (2025-01-02T03:15:25Z)
• Provably Efficient Information-Directed Sampling Algorithms for Multi-Agent Reinforcement Learning [50.92957910121088]
  This work designs and analyzes a novel set of algorithms for multi-agent reinforcement learning (MARL) based on the principle of information-directed sampling (IDS) For episodic two-player zero-sum MGs, we present three sample-efficient algorithms for learning Nash equilibrium. We extend Reg-MAIDS to multi-player general-sum MGs and prove that it can learn either the Nash equilibrium or coarse correlated equilibrium in a sample efficient manner.
  arXiv  Detail & Related papers  (2024-04-30T06:48:56Z)
• Doubly Robust Instance-Reweighted Adversarial Training [107.40683655362285]
  We propose a novel doubly-robust instance reweighted adversarial framework. Our importance weights are obtained by optimizing the KL-divergence regularized loss function. Our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance.
  arXiv  Detail & Related papers  (2023-08-01T06:16:18Z)
• An Information-Theoretic Analysis of Bayesian Reinforcement Learning [44.025369660607645]
  We specialize this definition to reinforcement learning problems modeled as Markov decision processes (MDPs) whose kernel parameters are unknown to the agent. We show that our bounds can recover from below the current information-theoretic bounds by Russo and Van Roy.
  arXiv  Detail & Related papers  (2022-07-18T16:28:01Z)
• Building Robust Ensembles via Margin Boosting [98.56381714748096]
  In adversarial robustness, a single model does not usually have enough power to defend against all possible adversarial attacks. We develop an algorithm for learning an ensemble with maximum margin. We show that our algorithm not only outperforms existing ensembling techniques, but also large models trained in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-06-07T14:55:58Z)
• Impact of Parameter Sparsity on Stochastic Gradient MCMC Methods for Bayesian Deep Learning [15.521736934292354]
  We investigate the potential of sparse network structures to flexibly trade-off storage costs and inference run time. We show that certain classes of randomly selected substructures can perform as well as substructures derived from state-of-the-art iterative pruning methods.
  arXiv  Detail & Related papers  (2022-02-08T10:34:05Z)
• Unifying Model Explainability and Robustness for Joint Text Classification and Rationale Extraction [11.878012909876713]
  We propose a joint classification and rationale extraction model named AT-BMC. It includes two key mechanisms: mixed Adversarial Training (AT) is designed to use various perturbations in discrete and embedding space to improve the model's robustness, and Boundary Match Constraint (BMC) helps to locate rationales more precisely with the guidance of boundary information. Performances on benchmark datasets demonstrate that the proposed AT-BMC outperforms baselines on both classification and rationale extraction by a large margin.
  arXiv  Detail & Related papers  (2021-12-20T09:48:32Z)
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning [0.0]
  The rampant adoption of ML methodologies has revealed that models are usually adopted to make decisions without taking into account the uncertainties in their predictions. We believe that developing ML systems that take into predictive account uncertainties and are robust against adversarial examples is a must for real-world tasks.
  arXiv  Detail & Related papers  (2021-09-25T23:02:47Z)
• Adversarial Robustness via Fisher-Rao Regularization [33.134075068748984]
  Adrial robustness has become a topic of growing interest in machine learning. Fire is a new Fisher-Rao regularization for the categorical cross-entropy loss.
  arXiv  Detail & Related papers  (2021-06-12T04:12:58Z)
• Unsupervised Scale-consistent Depth Learning from Video [131.3074342883371]
  We propose a monocular depth estimator SC-Depth, which requires only unlabelled videos for training. Thanks to the capability of scale-consistent prediction, we show that our monocular-trained deep networks are readily integrated into the ORB-SLAM2 system. The proposed hybrid Pseudo-RGBD SLAM shows compelling results in KITTI, and it generalizes well to the KAIST dataset without additional training.
  arXiv  Detail & Related papers  (2021-05-25T02:17:56Z)
• What Are Bayesian Neural Network Posteriors Really Like? [63.950151520585024]
  We show that Hamiltonian Monte Carlo can achieve significant performance gains over standard and deep ensembles. We also show that deep distributions are similarly close to HMC as standard SGLD, and closer than standard variational inference.
  arXiv  Detail & Related papers  (2021-04-29T15:38:46Z)
• A Hamiltonian Monte Carlo Method for Probabilistic Adversarial Attack and Learning [122.49765136434353]
  We present an effective method, called Hamiltonian Monte Carlo with Accumulated Momentum (HMCAM), aiming to generate a sequence of adversarial examples. We also propose a new generative method called Contrastive Adversarial Training (CAT), which approaches equilibrium distribution of adversarial examples. Both quantitative and qualitative analysis on several natural image datasets and practical systems have confirmed the superiority of the proposed algorithm.
  arXiv  Detail & Related papers  (2020-10-15T16:07:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.