CTBENCH: A Library and Benchmark for Certified Training

• URL: http://arxiv.org/abs/2406.04848v3
• Date: Mon, 03 Feb 2025 14:49:02 GMT
• Title: CTBENCH: A Library and Benchmark for Certified Training
• Authors: Yuhao Mao, Stefan Balauca, Martin Vechev,
• Abstract summary: We introduce CTBench, a high-quality benchmark for certified training.<n>We show that almost all algorithms in CTBench surpass the corresponding reported performance in literature.<n>We provide new insights into the current state of certified training.
• Score: 2.4829062265865764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training certifiably robust neural networks is an important but challenging task. While many algorithms for (deterministic) certified training have been proposed, they are often evaluated on different training schedules, certification methods, and systematically under-tuned hyperparameters, making it difficult to compare their performance. To address this challenge, we introduce CTBench, a unified library and a high-quality benchmark for certified training that evaluates all algorithms under fair settings and systematically tuned hyperparameters. We show that (1) almost all algorithms in CTBench surpass the corresponding reported performance in literature in the magnitude of algorithmic improvements, thus establishing new state-of-the-art, and (2) the claimed advantage of recent algorithms drops significantly when we enhance the outdated baselines with a fair training schedule, a fair certification method and well-tuned hyperparameters. Based on CTBench, we provide new insights into the current state of certified training, including (1) certified models have less fragmented loss surface, (2) certified models share many mistakes, (3) certified models have more sparse activations, (4) reducing regularization cleverly is crucial for certified training especially for large radii and (5) certified training has the potential to improve out-of-distribution generalization. We are confident that CTBench will serve as a benchmark and testbed for future research in certified training.

Related papers

• Adaptive Hierarchical Certification for Segmentation using Randomized Smoothing [87.48628403354351]
  certification for machine learning is proving that no adversarial sample can evade a model within a range under certain conditions. Common certification methods for segmentation use a flat set of fine-grained classes, leading to high abstain rates due to model uncertainty. We propose a novel, more practical setting, which certifies pixels within a multi-level hierarchy, and adaptively relaxes the certification to a coarser level for unstable components.
  arXiv  Detail & Related papers  (2024-02-13T11:59:43Z)
• Adaptive Certified Training: Towards Better Accuracy-Robustness Tradeoffs [17.46692880231195]
  We propose a novel certified training method based on a key insight that training with adaptive certified radii helps to improve the accuracy and robustness of the model. We demonstrate the effectiveness of the proposed method on MNIST, CIFAR-10, and TinyImageNet datasets.
  arXiv  Detail & Related papers  (2023-07-24T18:59:46Z)
• No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models [31.080446886440757]
  In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, dropping), batch selection (selective backprop, RHO loss), and efficient layers (Lion, Sophia) We find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables machines to be done on arbitrary computation by mapping all computation time to a reference machine which we call reference system time.
  arXiv  Detail & Related papers  (2023-07-12T20:10:14Z)
• Benchmarking Neural Network Training Algorithms [46.39165332979669]
  Training algorithms are an essential part of every deep learning pipeline. As a community, we are unable to reliably identify training algorithm improvements. We introduce a new, competitive, time-to-result benchmark using multiple workloads running on fixed hardware.
  arXiv  Detail & Related papers  (2023-06-12T15:21:02Z)
• Tight Certification of Adversarially Trained Neural Networks via Nonconvex Low-Rank Semidefinite Relaxations [12.589519278962378]
  We propose a non certification technique for adversarial network models. Non certification makes strong certifications comparable to much more expensive SDP methods, while optimizing variables dramatically fewer comparable to LP methods. Our experiments find that the non certification almost completely closes the gap towards exact certification adversarially trained models.
  arXiv  Detail & Related papers  (2022-11-30T18:46:00Z)
• A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks [65.34977803841007]
  Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. We show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one.
  arXiv  Detail & Related papers  (2022-11-16T00:11:04Z)
• Accelerating Certified Robustness Training via Knowledge Transfer [3.5934248574481717]
  We propose a framework for reducing the computational overhead of any certifiably robust training method through knowledge transfer. Our experiments on CIFAR-10 show that CRT speeds up certified robustness training by $8 times$ on average across three different architecture generations.
  arXiv  Detail & Related papers  (2022-10-25T19:12:28Z)
• Towards Diverse Evaluation of Class Incremental Learning: A Representation Learning Perspective [67.45111837188685]
  Class incremental learning (CIL) algorithms aim to continually learn new object classes from incrementally arriving data. We experimentally analyze neural network models trained by CIL algorithms using various evaluation protocols in representation learning.
  arXiv  Detail & Related papers  (2022-06-16T11:44:11Z)
• Smooth-Reduce: Leveraging Patches for Improved Certified Robustness [100.28947222215463]
  We propose a training-free, modified smoothing approach, Smooth-Reduce. Our algorithm classifies overlapping patches extracted from an input image, and aggregates the predicted logits to certify a larger radius around the input. We provide theoretical guarantees for such certificates, and empirically show significant improvements over other randomized smoothing methods.
  arXiv  Detail & Related papers  (2022-05-12T15:26:20Z)
• COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks [49.15885037760725]
  We focus on certifying the robustness of offline reinforcement learning (RL) in the presence of poisoning attacks. We propose the first certification framework, COPA, to certify the number of poisoning trajectories that can be tolerated. We prove that some of the proposed certification methods are theoretically tight and some are NP-Complete problems.
  arXiv  Detail & Related papers  (2022-03-16T05:02:47Z)
• Label, Verify, Correct: A Simple Few Shot Object Detection Method [93.84801062680786]
  We introduce a simple pseudo-labelling method to source high-quality pseudo-annotations from a training set. We present two novel methods to improve the precision of the pseudo-labelling process. Our method achieves state-of-the-art or second-best performance compared to existing approaches.
  arXiv  Detail & Related papers  (2021-12-10T18:59:06Z)
• An Automated Knowledge Mining and Document Classification System with Multi-model Transfer Learning [1.1852751647387592]
  Service manual documents are crucial to the engineering company as they provide guidelines and knowledge to service engineers. We propose an automated knowledge mining and document classification system with novel multi-model transfer learning approaches.
  arXiv  Detail & Related papers  (2021-06-24T03:03:46Z)
• Training ELECTRA Augmented with Multi-word Selection [53.77046731238381]
  We present a new text encoder pre-training method that improves ELECTRA based on multi-task learning. Specifically, we train the discriminator to simultaneously detect replaced tokens and select original tokens from candidate sets.
  arXiv  Detail & Related papers  (2021-05-31T23:19:00Z)
• Fast Certified Robust Training via Better Initialization and Shorter Warmup [95.81628508228623]
  We propose a new IBP and principled regularizers during the warmup stage to stabilize certified bounds. We find that batch normalization (BN) is a crucial architectural element to build best-performing networks for certified training.
  arXiv  Detail & Related papers  (2021-03-31T17:58:58Z)
• Fast Training of Provably Robust Neural Networks by SingleProp [71.19423596238568]
  We develop a new regularizer that is both more efficient than existing certified defenses. We demonstrate improvements in training speed and comparable certified accuracy compared to state-of-the-art certified defenses.
  arXiv  Detail & Related papers  (2021-02-01T22:12:51Z)
• Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free [115.81899803240758]
  Adversarial training and its many variants substantially improve deep network robustness, yet at the cost of compromising standard accuracy. This paper asks how to quickly calibrate a trained model in-situ, to examine the achievable trade-offs between its standard and robust accuracies. Our proposed framework, Once-for-all Adversarial Training (OAT), is built on an innovative model-conditional training framework.
  arXiv  Detail & Related papers  (2020-10-22T16:06:34Z)
• Regularized Training and Tight Certification for Randomized Smoothed Classifier with Provable Robustness [15.38718018477333]
  We derive a new regularized risk, in which the regularizer can adaptively encourage the accuracy and robustness of the smoothed counterpart. We also design a new certification algorithm, which can leverage the regularization effect to provide tighter robustness lower bound that holds with high probability.
  arXiv  Detail & Related papers  (2020-02-17T20:54:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.