Improving Certified Robustness via Adversarial Distillation

• URL: http://arxiv.org/abs/2606.31653v1
• Date: Tue, 30 Jun 2026 13:31:35 GMT
• Title: Improving Certified Robustness via Adversarial Distillation
• Abstract summary: We introduce AD-CERT, a certified training objective that combines adversarial distillation with an IBP upper bound.<n>We show that AD-CERT achieves state-of-the-art certified performance on several benchmarks.
• Score: 2.969291853301593
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Certified training aims to produce models whose predictions can be formally verified against adversarial perturbations, typically by optimising upper bounds on the worst-case loss over an allowed perturbation set. For neural networks, certified training methods based purely on tight relaxation bounds produce networks that are amenable to certification, but sacrifice standard accuracy. Conversely, adversarial training often yields stronger empirical robustness and standard accuracy, but the resulting models are generally difficult to certify with neural network verifiers. Recently, the literature has shown that better standard-certified accuracy trade-offs can be achieved by combining adversarial training objectives with loose over-approximations based on Interval Bound Propagation (IBP), effectively interpolating between lower and upper bounds of the worst-case loss. Building on this, we introduce AD-CERT, a certified training objective that combines adversarial distillation with an IBP upper bound. We show that distilling adversarial information over the logit space from an empirically robust teacher provides an effective lower bound surrogate for certified training, with AD-CERT achieving state-of-the-art certified performance on several robustness benchmarks. Furthermore, in a unified setup, distilling adversarial information at the logit-level is shown to improve certified accuracy over a robust feature-space distillation objective by up to 5.40 percentage points.

Related papers

• Veriphi: Attack-Guided Neural Network Verification with Dataset-Dependent Training Methods [0.0]
  We present a GPU-accelerated neural network verification system that combines fast adversarial attacks with formal bound certification.<n>We demonstrate that training method effectiveness is fundamentally dataset-dependent.<n>We achieve 5x verification speedup through attack-guided falsification and scale our approach to production-size models.
  arXiv  Detail & Related papers  (2026-06-16T20:02:51Z)
• Rethinking Evaluation Paradigms in IBP-based Certified Training [7.29878846692057]
  Deep neural networks achieve strong performance on many supervised learning tasks but remain vulnerable to adversarial perturbations.<n>To mitigate this, certified training techniques optimise for verifiable robustness during training.<n>We show that prior advancements are less pronounced than assumed and revealing previously unreported performance complementarities.
  arXiv  Detail & Related papers  (2026-06-01T12:00:15Z)
• The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation [67.26315138466312]
  We propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts.<n>Our findings highlight that capability distillation does not imply confidence, and that confidence should be treated as an essential objective in post-training.
  arXiv  Detail & Related papers  (2026-04-18T04:43:40Z)
• Learning Better Certified Models from Empirically-Robust Teachers [53.51898477987441]
  Adrial training attains strong empirical robustness to specific adversarial attacks by training on concrete adversarial perturbations.<n>It produces neural networks that are not amenable to strong robustness certificates through neural network verification.<n>Earlier certified training schemes directly train on bounds from network relaxations to obtain models that are certifiably robust, but display sub-par standard performance.
  arXiv  Detail & Related papers  (2026-02-02T16:30:53Z)
• Improved Certified Defenses against Data Poisoning with (Deterministic) Finite Aggregation [122.83280749890078]
  We propose an improved certified defense against general poisoning attacks, namely Finite Aggregation. In contrast to DPA, which directly splits the training set into disjoint subsets, our method first splits the training set into smaller disjoint subsets. We offer an alternative view of our method, bridging the designs of deterministic and aggregation-based certified defenses.
  arXiv  Detail & Related papers  (2022-02-05T20:08:58Z)
• How Robust are Randomized Smoothing based Defenses to Data Poisoning? [66.80663779176979]
  We present a previously unrecognized threat to robust machine learning models that highlights the importance of training-data quality. We propose a novel bilevel optimization-based data poisoning attack that degrades the robustness guarantees of certifiably robust classifiers. Our attack is effective even when the victim trains the models from scratch using state-of-the-art robust training methods.
  arXiv  Detail & Related papers  (2020-12-02T15:30:21Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)
• Certified Distributional Robustness on Smoothed Classifiers [27.006844966157317]
  We propose the worst-case adversarial loss over input distributions as a robustness certificate. By exploiting duality and the smoothness property, we provide an easy-to-compute upper bound as a surrogate for the certificate.
  arXiv  Detail & Related papers  (2020-10-21T13:22:25Z)
• Certifiably Adversarially Robust Detection of Out-of-Distribution Data [111.67388500330273]
  We aim for certifiable worst case guarantees for OOD detection by enforcing low confidence at the OOD point. We show that non-trivial bounds on the confidence for OOD data generalizing beyond the OOD dataset seen at training time are possible.
  arXiv  Detail & Related papers  (2020-07-16T17:16:47Z)
• Regularized Training and Tight Certification for Randomized Smoothed Classifier with Provable Robustness [15.38718018477333]
  We derive a new regularized risk, in which the regularizer can adaptively encourage the accuracy and robustness of the smoothed counterpart. We also design a new certification algorithm, which can leverage the regularization effect to provide tighter robustness lower bound that holds with high probability.
  arXiv  Detail & Related papers  (2020-02-17T20:54:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.