DE-CROP: Data-efficient Certified Robustness for Pretrained Classifiers
- URL: http://arxiv.org/abs/2210.08929v1
- Date: Mon, 17 Oct 2022 10:41:18 GMT
- Title: DE-CROP: Data-efficient Certified Robustness for Pretrained Classifiers
- Authors: Gaurav Kumar Nayak, Ruchit Rawal, Anirban Chakraborty
- Abstract summary: We propose a novel way to certify the robustness of pretrained models using only a few training samples.
Our proposed approach generates class-boundary and interpolated samples corresponding to each training sample.
We obtain significant improvements over the baseline on multiple benchmark datasets and also report similar performance under the challenging black box setup.
- Score: 21.741026088202126
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Certified defense using randomized smoothing is a popular technique to
provide robustness guarantees for deep neural networks against l2 adversarial
attacks. Existing works use this technique to provably secure a pretrained
non-robust model by training a custom denoiser network on entire training data.
However, access to the training set may be restricted to a handful of data
samples due to constraints such as high transmission cost and the proprietary
nature of the data. Thus, we formulate a novel problem of "how to certify the
robustness of pretrained models using only a few training samples". We observe
that training the custom denoiser directly using the existing techniques on
limited samples yields poor certification. To overcome this, our proposed
approach (DE-CROP) generates class-boundary and interpolated samples
corresponding to each training sample, ensuring high diversity in the feature
space of the pretrained classifier. We train the denoiser by maximizing the
similarity between the denoised output of the generated sample and the original
training sample in the classifier's logit space. We also perform distribution
level matching using domain discriminator and maximum mean discrepancy that
yields further benefit. In white box setup, we obtain significant improvements
over the baseline on multiple benchmark datasets and also report similar
performance under the challenging black box setup.
Related papers
- CPSample: Classifier Protected Sampling for Guarding Training Data During Diffusion [58.64822817224639]
Diffusion models have a tendency to exactly replicate their training data, especially when trained on small datasets.
We present CPSample, a method that modifies the sampling process to prevent training data replication while preserving image quality.
CPSample achieves FID scores of 4.97 and 2.97 on CIFAR-10 and CelebA-64, respectively, without producing exact replicates of the training data.
arXiv Detail & Related papers (2024-09-11T05:42:01Z) - CALICO: Confident Active Learning with Integrated Calibration [11.978551396144532]
We propose an AL framework that self-calibrates the confidence used for sample selection during the training process.
We show improved classification performance compared to a softmax-based classifier with fewer labeled samples.
arXiv Detail & Related papers (2024-07-02T15:05:19Z) - Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice.
We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
arXiv Detail & Related papers (2023-12-27T09:03:43Z) - Learning from Data with Noisy Labels Using Temporal Self-Ensemble [11.245833546360386]
Deep neural networks (DNNs) have an enormous capacity to memorize noisy labels.
Current state-of-the-art methods present a co-training scheme that trains dual networks using samples associated with small losses.
We propose a simple yet effective robust training scheme that operates by training only a single network.
arXiv Detail & Related papers (2022-07-21T08:16:31Z) - Distributed Adversarial Training to Robustify Deep Neural Networks at
Scale [100.19539096465101]
Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification.
To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training.
We propose a large-batch adversarial training framework implemented over multiple machines.
arXiv Detail & Related papers (2022-06-13T15:39:43Z) - CAFA: Class-Aware Feature Alignment for Test-Time Adaptation [50.26963784271912]
Test-time adaptation (TTA) aims to address this challenge by adapting a model to unlabeled data at test time.
We propose a simple yet effective feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which simultaneously encourages a model to learn target representations in a class-discriminative manner.
arXiv Detail & Related papers (2022-06-01T03:02:07Z) - A Data Cartography based MixUp for Pre-trained Language Models [47.90235939359225]
MixUp is a data augmentation strategy where additional samples are generated during training by combining random pairs of training samples and their labels.
We propose TDMixUp, a novel MixUp strategy that leverages Training Dynamics and allows more informative samples to be combined for generating new data samples.
We empirically validate that our method not only achieves competitive performance using a smaller subset of the training data compared with strong baselines, but also yields lower expected calibration error on the pre-trained language model, BERT, on both in-domain and out-of-domain settings in a wide range of NLP tasks.
arXiv Detail & Related papers (2022-05-06T17:59:19Z) - Deep Ensembles for Low-Data Transfer Learning [21.578470914935938]
We study different ways of creating ensembles from pre-trained models.
We show that the nature of pre-training itself is a performant source of diversity.
We propose a practical algorithm that efficiently identifies a subset of pre-trained models for any downstream dataset.
arXiv Detail & Related papers (2020-10-14T07:59:00Z) - Improved Robustness to Open Set Inputs via Tempered Mixup [37.98372874213471]
We propose a simple regularization technique that improves open set robustness without a background dataset.
Our method achieves state-of-the-art results on open set classification baselines and easily scales to large-scale open set classification problems.
arXiv Detail & Related papers (2020-09-10T04:01:31Z) - Pre-training Is (Almost) All You Need: An Application to Commonsense
Reasoning [61.32992639292889]
Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks.
We introduce a new scoring method that casts a plausibility ranking task in a full-text format.
We show that our method provides a much more stable training phase across random restarts.
arXiv Detail & Related papers (2020-04-29T10:54:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.