CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection

• URL: http://arxiv.org/abs/2405.16417v1
• Date: Sun, 26 May 2024 03:28:59 GMT
• Title: CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection
• Authors: Lin Zhu, Yifeng Yang, Qinying Gu, Xinbing Wang, Chenghu Zhou, Nanyang Ye,
• Abstract summary: We show that minimizing the magnitude of energy scores on training data leads to domain-consistent Hessians of classification loss. We have developed a unified fine-tuning framework that allows for concurrent optimization of both tasks.
• Score: 42.33618249731874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent vision-language pre-trained models (VL-PTMs) have shown remarkable success in open-vocabulary tasks. However, downstream use cases often involve further fine-tuning of VL-PTMs, which may distort their general knowledge and impair their ability to handle distribution shifts. In real-world scenarios, machine learning systems inevitably encounter both covariate shifts (e.g., changes in image styles) and semantic shifts (e.g., test-time unseen classes). This highlights the importance of enhancing out-of-distribution (OOD) generalization on covariate shifts and simultaneously detecting semantic-shifted unseen classes. Thus a critical but underexplored question arises: How to improve VL-PTMs' generalization ability to closed-set OOD data, while effectively detecting open-set unseen classes during fine-tuning? In this paper, we propose a novel objective function of OOD detection that also serves to improve OOD generalization. We show that minimizing the gradient magnitude of energy scores on training data leads to domain-consistent Hessians of classification loss, a strong indicator for OOD generalization revealed by theoretical analysis. Based on this finding, we have developed a unified fine-tuning framework that allows for concurrent optimization of both tasks. Extensive experiments have demonstrated the superiority of our method. The code is available at https://github.com/LinLLLL/CRoFT.

Related papers

• WeiPer: OOD Detection using Weight Perturbations of Class Projections [11.130659240045544]
  We introduce perturbations of the class projections in the final fully connected layer which creates a richer representation of the input. We achieve state-of-the-art OOD detection results across multiple benchmarks of the OpenOOD framework.
  arXiv  Detail & Related papers  (2024-05-27T13:38:28Z)
• Towards Robust Out-of-Distribution Generalization Bounds via Sharpness [41.65692353665847]
  We study the effect of sharpness on how a model tolerates data change in domain shift. We propose a sharpness-based OOD generalization bound by taking robustness into consideration.
  arXiv  Detail & Related papers  (2024-03-11T02:57:27Z)
• Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization [11.140366256534474]
  Existing vision-language models exhibit strong generalization on a variety of visual domains and tasks. We propose a novel approach OGEN to improve the OOD GENeralization of finetuned models. Specifically, a class-conditional feature generator is introduced to synthesize OOD features using just the class name of any unknown class.
  arXiv  Detail & Related papers  (2024-01-29T06:57:48Z)
• TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization [89.54947228958494]
  This paper focuses on the fine-tuning of an adversarially pre-trained model in various classification tasks. We propose a novel statistics-based approach, Two-WIng NormliSation (TWINS) fine-tuning framework. TWINS is shown to be effective on a wide range of image classification datasets in terms of both generalization and robustness.
  arXiv  Detail & Related papers  (2023-03-20T14:12:55Z)
• Energy-based Out-of-Distribution Detection for Graph Neural Networks [76.0242218180483]
  We propose a simple, powerful and efficient OOD detection model for GNN-based learning on graphs, which we call GNNSafe. GNNSafe achieves up to $17.0%$ AUROC improvement over state-of-the-arts and it could serve as simple yet strong baselines in such an under-developed area.
  arXiv  Detail & Related papers  (2023-02-06T16:38:43Z)
• CLIPood: Generalizing CLIP to Out-of-Distributions [73.86353105017076]
  Contrastive language-image pre-training (CLIP) models have shown impressive zero-shot ability, but the further adaptation of CLIP on downstream tasks undesirably degrades OOD performances. We propose CLIPood, a fine-tuning method that can adapt CLIP models to OOD situations where both domain shifts and open classes may occur on unseen test data. Experiments on diverse datasets with different OOD scenarios show that CLIPood consistently outperforms existing generalization techniques.
  arXiv  Detail & Related papers  (2023-02-02T04:27:54Z)
• Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priors [17.936426699670864]
  We show that sample-wise Adversarial Training (AT) has limited improvement on Out-of-Distribution (OOD) generalization. We propose two AT variants with low-rank structures to train OOD-robust models. Our proposed approaches outperform Empirical Risk Minimization (ERM) and sample-wise AT.
  arXiv  Detail & Related papers  (2022-10-13T07:37:42Z)
• Improved OOD Generalization via Conditional Invariant Regularizer [43.62211060412388]
  We show that given a class label, conditionally independent models of spurious attributes are OOD general. Based on this, metric Conditional Variation (CSV) which controls OOD error is proposed to measure such conditional independence. An algorithm with minicave convergence rate is proposed to solve the problem.
  arXiv  Detail & Related papers  (2022-07-14T06:34:21Z)
• On the Practicality of Deterministic Epistemic Uncertainty [106.06571981780591]
  deterministic uncertainty methods (DUMs) achieve strong performance on detecting out-of-distribution data. It remains unclear whether DUMs are well calibrated and can seamlessly scale to real-world applications.
  arXiv  Detail & Related papers  (2021-07-01T17:59:07Z)
• Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLU [70.44344060176952]
  Intent classification is a major task in spoken language understanding (SLU) Recent works have shown that using extra data and labels can improve the OOD detection performance. This paper proposes to train a model with only IND data while supporting both IND intent classification and OOD detection.
  arXiv  Detail & Related papers  (2021-06-28T08:27:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.