CTBENCH: A Library and Benchmark for Certified Training

• URL: http://arxiv.org/abs/2406.04848v3
• Date: Mon, 03 Feb 2025 14:49:02 GMT
• Title: CTBENCH: A Library and Benchmark for Certified Training
• Authors: Yuhao Mao, Stefan Balauca, Martin Vechev,
• Abstract summary: We introduce CTBench, a high-quality benchmark for certified training. We show that almost all algorithms in CTBench surpass the corresponding reported performance in literature. We provide new insights into the current state of certified training.
• Score: 2.4829062265865764
• License:
• Abstract: Training certifiably robust neural networks is an important but challenging task. While many algorithms for (deterministic) certified training have been proposed, they are often evaluated on different training schedules, certification methods, and systematically under-tuned hyperparameters, making it difficult to compare their performance. To address this challenge, we introduce CTBench, a unified library and a high-quality benchmark for certified training that evaluates all algorithms under fair settings and systematically tuned hyperparameters. We show that (1) almost all algorithms in CTBench surpass the corresponding reported performance in literature in the magnitude of algorithmic improvements, thus establishing new state-of-the-art, and (2) the claimed advantage of recent algorithms drops significantly when we enhance the outdated baselines with a fair training schedule, a fair certification method and well-tuned hyperparameters. Based on CTBench, we provide new insights into the current state of certified training, including (1) certified models have less fragmented loss surface, (2) certified models share many mistakes, (3) certified models have more sparse activations, (4) reducing regularization cleverly is crucial for certified training especially for large radii and (5) certified training has the potential to improve out-of-distribution generalization. We are confident that CTBench will serve as a benchmark and testbed for future research in certified training.

Related papers

• The Surprising Effectiveness of Test-Time Training for Abstract Reasoning [64.36534512742736]
  We investigate the effectiveness of test-time training (TTT) as a mechanism for improving models' reasoning capabilities. TTT significantly improves performance on ARC tasks, achieving up to 6x improvement in accuracy compared to base fine-tuned models. Our findings suggest that explicit symbolic search is not the only path to improved abstract reasoning in neural language models.
  arXiv  Detail & Related papers  (2024-11-11T18:59:45Z)
• Adaptive Hierarchical Certification for Segmentation using Randomized Smoothing [87.48628403354351]
  certification for machine learning is proving that no adversarial sample can evade a model within a range under certain conditions. Common certification methods for segmentation use a flat set of fine-grained classes, leading to high abstain rates due to model uncertainty. We propose a novel, more practical setting, which certifies pixels within a multi-level hierarchy, and adaptively relaxes the certification to a coarser level for unstable components.
  arXiv  Detail & Related papers  (2024-02-13T11:59:43Z)
• Adaptive Certified Training: Towards Better Accuracy-Robustness Tradeoffs [17.46692880231195]
  We propose a novel certified training method based on a key insight that training with adaptive certified radii helps to improve the accuracy and robustness of the model. We demonstrate the effectiveness of the proposed method on MNIST, CIFAR-10, and TinyImageNet datasets.
  arXiv  Detail & Related papers  (2023-07-24T18:59:46Z)
• Tight Certification of Adversarially Trained Neural Networks via Nonconvex Low-Rank Semidefinite Relaxations [12.589519278962378]
  We propose a non certification technique for adversarial network models. Non certification makes strong certifications comparable to much more expensive SDP methods, while optimizing variables dramatically fewer comparable to LP methods. Our experiments find that the non certification almost completely closes the gap towards exact certification adversarially trained models.
  arXiv  Detail & Related papers  (2022-11-30T18:46:00Z)
• A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks [65.34977803841007]
  Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. We show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one.
  arXiv  Detail & Related papers  (2022-11-16T00:11:04Z)
• Accelerating Certified Robustness Training via Knowledge Transfer [3.5934248574481717]
  We propose a framework for reducing the computational overhead of any certifiably robust training method through knowledge transfer. Our experiments on CIFAR-10 show that CRT speeds up certified robustness training by $8 times$ on average across three different architecture generations.
  arXiv  Detail & Related papers  (2022-10-25T19:12:28Z)
• Smooth-Reduce: Leveraging Patches for Improved Certified Robustness [100.28947222215463]
  We propose a training-free, modified smoothing approach, Smooth-Reduce. Our algorithm classifies overlapping patches extracted from an input image, and aggregates the predicted logits to certify a larger radius around the input. We provide theoretical guarantees for such certificates, and empirically show significant improvements over other randomized smoothing methods.
  arXiv  Detail & Related papers  (2022-05-12T15:26:20Z)
• An Automated Knowledge Mining and Document Classification System with Multi-model Transfer Learning [1.1852751647387592]
  Service manual documents are crucial to the engineering company as they provide guidelines and knowledge to service engineers. We propose an automated knowledge mining and document classification system with novel multi-model transfer learning approaches.
  arXiv  Detail & Related papers  (2021-06-24T03:03:46Z)
• Fast Certified Robust Training via Better Initialization and Shorter Warmup [95.81628508228623]
  We propose a new IBP and principled regularizers during the warmup stage to stabilize certified bounds. We find that batch normalization (BN) is a crucial architectural element to build best-performing networks for certified training.
  arXiv  Detail & Related papers  (2021-03-31T17:58:58Z)
• Fast Training of Provably Robust Neural Networks by SingleProp [71.19423596238568]
  We develop a new regularizer that is both more efficient than existing certified defenses. We demonstrate improvements in training speed and comparable certified accuracy compared to state-of-the-art certified defenses.
  arXiv  Detail & Related papers  (2021-02-01T22:12:51Z)
• Regularized Training and Tight Certification for Randomized Smoothed Classifier with Provable Robustness [15.38718018477333]
  We derive a new regularized risk, in which the regularizer can adaptively encourage the accuracy and robustness of the smoothed counterpart. We also design a new certification algorithm, which can leverage the regularization effect to provide tighter robustness lower bound that holds with high probability.
  arXiv  Detail & Related papers  (2020-02-17T20:54:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.