Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performance

• URL: http://arxiv.org/abs/2509.05328v1
• Date: Sun, 31 Aug 2025 12:14:34 GMT
• Title: Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performance
• Authors: Xiang Yuan, Jun Shu, Deyu meng, Zongben Xu,
• Abstract summary: We propose a novel regularization that constrains the distance of fine-tuning and pre-trained model in the function space with simulated OOD samples.<n>Our approach could consistently improve both downstream task ID fine-tuning performance and OOD robustness across a variety of CLIP backbones.
• Score: 72.57668440744301
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robust fine-tuning aims to achieve competitive in-distribution (ID) performance while maintaining the out-of-distribution (OOD) robustness of a pre-trained model when transferring it to a downstream task. To remedy this, most robust fine-tuning methods aim to preserve the pretrained weights, features, or logits. However, we find that these methods cannot always improve OOD robustness for different model architectures. This is due to the OOD robustness requiring the model function to produce stable prediction for input information of downstream tasks, while existing methods might serve as a poor proxy for the optimization in the function space. Based on this finding, we propose a novel regularization that constrains the distance of fine-tuning and pre-trained model in the function space with the simulated OOD samples, aiming to preserve the OOD robustness of the pre-trained model. Besides, to further enhance the OOD robustness capability of the fine-tuning model, we introduce an additional consistency regularization to promote stable predictions of perturbed samples. Extensive experiments demonstrate our approach could consistently improve both downstream task ID fine-tuning performance and OOD robustness across a variety of CLIP backbones, outperforming existing regularization-based robust fine-tuning methods.

Related papers

• Contrastive Knowledge Transfer and Robust Optimization for Secure Alignment of Large Language Models [9.353236468990945]
  This paper addresses the limitations of large-scale language models in safety alignment and robustness.<n>It proposes a fine-tuning method that combines contrastive distillation with noise-robust training.<n>Results show that the method significantly outperforms existing baselines in knowledge transfer, robustness, and overall safety.
  arXiv  Detail & Related papers  (2025-10-31T00:54:33Z)
• MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics [72.00014675808228]
  Instability in Large Language Models evaluation process obscures true learning dynamics.<n>We introduce textbfMaP, a framework that integrates underlineMerging underlineand the underlinePass@k metric.<n>Experiments show that MaP yields significantly smoother performance curves, reduces inter-run variance, and ensures more consistent rankings.
  arXiv  Detail & Related papers  (2025-10-10T11:40:27Z)
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection [84.78366452133514]
  Human-Object Interaction (HOI) detection is crucial for robot-human assistance, enabling context-aware support.<n>We introduce the first benchmark for HOI detection, evaluating model resilience under diverse challenges.<n>Our benchmark, RoHOI, includes 20 corruption types based on the HICO-DET and V-COCO datasets and a new robustness-focused metric.
  arXiv  Detail & Related papers  (2025-07-12T01:58:04Z)
• Robust Fine-tuning of Zero-shot Models via Variance Reduction [56.360865951192324]
  When fine-tuning zero-shot models, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD) We propose a sample-wise ensembling technique that can simultaneously attain the best ID and OOD accuracy without the trade-offs.
  arXiv  Detail & Related papers  (2024-11-11T13:13:39Z)
• Large Pre-Training Datasets Don't Always Guarantee Robustness after Fine-Tuning [29.69990405081772]
  We propose the Robustness Inheritance Benchmark (ImageNet-RIB) to assess robustness preservation in fine-tuned models.<n>We find that fine-tuning reduces robustness across pretrained models.<n>Models pretrained on the largest and most diverse datasets exhibit both larger robustness losses and lower absolute robustness after fine-tuning on small datasets.
  arXiv  Detail & Related papers  (2024-10-28T22:33:22Z)
• Robust Fine-tuning for Pre-trained 3D Point Cloud Models [15.404188754049317]
  This paper presents a robust fine-tuning method designed for pre-trained 3D point cloud models. We highlight the limitations of current fine-tuning methods and the challenges of learning robust models. Experimental results demonstrate the effectiveness of WiSE-FT-LP in enhancing model robustness.
  arXiv  Detail & Related papers  (2024-04-25T08:52:25Z)
• Towards Calibrated Robust Fine-Tuning of Vision-Language Models [97.19901765814431]
  This work proposes a robust fine-tuning method that improves both OOD accuracy and confidence calibration simultaneously in vision language models. We show that both OOD classification and OOD calibration errors have a shared upper bound consisting of two terms of ID data. Based on this insight, we design a novel framework that conducts fine-tuning with a constrained multimodal contrastive loss enforcing a larger smallest singular value.
  arXiv  Detail & Related papers  (2023-11-03T05:41:25Z)
• Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularization [53.04697800214848]
  Adversarial Training (AT) is pivotal in fortifying the robustness of deep learning models. AT methods, relying on direct iterative updates for target model's defense, frequently encounter obstacles such as unstable training and catastrophic overfitting. We present a general proxy guided defense framework, LAST' (bf Learn from the Pbf ast)
  arXiv  Detail & Related papers  (2023-10-19T13:13:41Z)
• Models Out of Line: A Fourier Lens on Distribution Shift Robustness [29.12208822285158]
  Improving accuracy of deep neural networks (DNNs) on out-of-distribution (OOD) data is critical to an acceptance of deep learning (DL) in real world applications. Recently, some promising approaches have been developed to improve OOD robustness. There still is no clear understanding of the conditions on OOD data and model properties that are required to observe effective robustness.
  arXiv  Detail & Related papers  (2022-07-08T18:05:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.