DAFT: Distilling Adversarially Fine-tuned Models for Better OOD Generalization

• URL: http://arxiv.org/abs/2208.09139v1
• Date: Fri, 19 Aug 2022 03:48:17 GMT
• Title: DAFT: Distilling Adversarially Fine-tuned Models for Better OOD Generalization
• Authors: Anshul Nasery, Sravanti Addepalli, Praneeth Netrapalli, Prateek Jain
• Abstract summary: We consider the problem of OOD generalization, where the goal is to train a model that performs well on test distributions that are different from the training distribution. We propose a new method - DAFT - based on the intuition that adversarially robust combination of a large number of rich features should provide OOD robustness. We evaluate DAFT on standard benchmarks in the DomainBed framework, and demonstrate that DAFT achieves significant improvements over the current state-of-the-art OOD generalization methods.
• Score: 35.53270942633211
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We consider the problem of OOD generalization, where the goal is to train a model that performs well on test distributions that are different from the training distribution. Deep learning models are known to be fragile to such shifts and can suffer large accuracy drops even for slightly different test distributions. We propose a new method - DAFT - based on the intuition that adversarially robust combination of a large number of rich features should provide OOD robustness. Our method carefully distills the knowledge from a powerful teacher that learns several discriminative features using standard training while combining them using adversarial training. The standard adversarial training procedure is modified to produce teachers which can guide the student better. We evaluate DAFT on standard benchmarks in the DomainBed framework, and demonstrate that DAFT achieves significant improvements over the current state-of-the-art OOD generalization methods. DAFT consistently out-performs well-tuned ERM and distillation baselines by up to 6%, with more pronounced gains for smaller networks.

Related papers

• Faithful Label-free Knowledge Distillation [8.572967695281054]
  This paper presents a label-free knowledge distillation approach called Teacher in the Middle (TinTeM) It produces a more faithful student, which better replicates the behavior of the teacher network across a range of benchmarks testing model robustness, generalisability and out-of-distribution detection.
  arXiv  Detail & Related papers  (2024-11-22T01:48:44Z)
• Towards Fairness-Aware Adversarial Learning [13.932705960012846]
  We propose a novel learning paradigm, named Fairness-Aware Adversarial Learning (FAAL) Our method aims to find the worst distribution among different categories, and the solution is guaranteed to obtain the upper bound performance with high probability. In particular, FAAL can fine-tune an unfair robust model to be fair within only two epochs, without compromising the overall clean and robust accuracies.
  arXiv  Detail & Related papers  (2024-02-27T18:01:59Z)
• Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Off [107.35833747750446]
  adversarial examples can be crafted by adding imperceptible perturbations to legitimate documents. This vulnerability raises significant concerns about their reliability and hinders the widespread deployment of NRMs. In this study, we establish theoretical guarantees regarding the effectiveness-robustness trade-off in NRMs.
  arXiv  Detail & Related papers  (2023-12-16T05:38:39Z)
• Towards Calibrated Robust Fine-Tuning of Vision-Language Models [97.19901765814431]
  This work proposes a robust fine-tuning method that improves both OOD accuracy and confidence calibration simultaneously in vision language models. We show that both OOD classification and OOD calibration errors have a shared upper bound consisting of two terms of ID data. Based on this insight, we design a novel framework that conducts fine-tuning with a constrained multimodal contrastive loss enforcing a larger smallest singular value.
  arXiv  Detail & Related papers  (2023-11-03T05:41:25Z)
• On the Robustness of Open-World Test-Time Training: Self-Training with Dynamic Prototype Expansion [46.30241353155658]
  Generalizing deep learning models to unknown target domain distribution with low latency has motivated research into test-time training/adaptation (TTT/TTA) Many state-of-the-art methods fail to maintain the performance when the target domain is contaminated with strong out-of-distribution (OOD) data. We develop an adaptive strong OOD pruning which improves the efficacy of the self-training TTT method. We regularize self-training with distribution alignment and the combination yields the state-of-the-art performance on 5 OWTTT benchmarks.
  arXiv  Detail & Related papers  (2023-08-19T08:27:48Z)
• Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation [12.39860047886679]
  Adversarial Training is a practical approach for improving the robustness of deep neural networks against adversarial attacks. We introduce Balanced Multi-Teacher Adversarial Robustness Distillation (B-MTARD) to guide the model's Adversarial Training process. B-MTARD outperforms the state-of-the-art methods against various adversarial attacks.
  arXiv  Detail & Related papers  (2023-06-28T12:47:01Z)
• A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking [54.89987482509155]
  robustness of deep neural networks is usually lacking under adversarial examples, common corruptions, and distribution shifts. We establish a comprehensive benchmark robustness called textbfARES-Bench on the image classification task. By designing the training settings accordingly, we achieve the new state-of-the-art adversarial robustness.
  arXiv  Detail & Related papers  (2023-02-28T04:26:20Z)
• Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free [115.81899803240758]
  Adversarial training and its many variants substantially improve deep network robustness, yet at the cost of compromising standard accuracy. This paper asks how to quickly calibrate a trained model in-situ, to examine the achievable trade-offs between its standard and robust accuracies. Our proposed framework, Once-for-all Adversarial Training (OAT), is built on an innovative model-conditional training framework.
  arXiv  Detail & Related papers  (2020-10-22T16:06:34Z)
• Adversarial Robustness on In- and Out-Distribution Improves Explainability [109.68938066821246]
  RATIO is a training procedure for robustness via Adversarial Training on In- and Out-distribution. RATIO achieves state-of-the-art $l$-adrial on CIFAR10 and maintains better clean accuracy.
  arXiv  Detail & Related papers  (2020-03-20T18:57:52Z)
• Adversarial Distributional Training for Robust Deep Learning [53.300984501078126]
  Adversarial training (AT) is among the most effective techniques to improve model robustness by augmenting training data with adversarial examples. Most existing AT methods adopt a specific attack to craft adversarial examples, leading to the unreliable robustness against other unseen attacks. In this paper, we introduce adversarial distributional training (ADT), a novel framework for learning robust models.
  arXiv  Detail & Related papers  (2020-02-14T12:36:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.