The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models

• URL: http://arxiv.org/abs/2603.27139v2
• Date: Fri, 03 Apr 2026 23:58:57 GMT
• Title: The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models
• Abstract summary: Generalization-preserving methods retain ID/OOD performance but leave models vulnerable to adversarial attacks.<n>Our key insight is that the robustness trade-off stems from two geometric failures: sharp, anisotropic minima in parameter space and unstable feature representations that deform under perturbation.<n>We propose GRACE, a unified fine-tuning framework that jointly regularizes the parameter-space curvature and feature-space invariance for VLMs.
• Score: 33.10473765967503
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning approaches for Vision-Language Models (VLMs) face a critical three-way trade-off between In-Distribution (ID) accuracy, Out-of-Distribution (OOD) generalization, and adversarial robustness. Existing robust fine-tuning strategies resolve at most two axes of this trade-off. Generalization-preserving methods retain ID/OOD performance but leave models vulnerable to adversarial attacks, while adversarial training improves robustness to targeted attacks but degrades ID/OOD accuracy. Our key insight is that the robustness trade-off stems from two geometric failures: sharp, anisotropic minima in parameter space and unstable feature representations that deform under perturbation. To address this, we propose GRACE (Gram-aligned Robustness via Adaptive Curvature Estimation), a unified fine-tuning framework that jointly regularizes the parameter-space curvature and feature-space invariance for VLMs. Grounded in Robust PAC-Bayes theory, GRACE employs adaptive weight perturbations scaled by local curvature to promote flatter minima, combined with a feature alignment loss that maintains representation consistency across clean, adversarial, and OOD inputs. On ImageNet fine-tuning of CLIP models, GRACE simultaneously improves ID accuracy by 10.8%, and adversarial accuracy by 13.5% while maintaining 57.0% OOD accuracy (vs. 57.4% zero-shot baseline). Geometric analysis confirms that GRACE converges to flatter minima without feature distortion across distribution shifts, providing a principled step toward generalized robustness in foundation VLMs.

Related papers

• Deformable Gaussian Occupancy: Decoupling Rigid and Nonrigid Motion with Factorized Distillation [81.44955493339835]
  DeGO is a deformable Gaussian occupancy framework that unifies decoupled Gaussian deformation with factorized 4D foundation-model distillation.<n> Experiments on the Occ3D-NuScenes benchmark demonstrate that our method achieves state-of-the-art performance under weak supervision.
  arXiv  Detail & Related papers  (2026-05-27T15:09:55Z)
• AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models [38.50355675955809]
  AGC achieves superior performance across eight fine-grained datasets and three CLIP backbones.<n>Our findings reveal a fundamental geometric property of CLIP features, offering a highly efficient and effective paradigm for robust multimodal deployment.
  arXiv  Detail & Related papers  (2026-05-15T03:48:22Z)
• Segment Anything with Robust Uncertainty-Accuracy Correlation [38.06380944319879]
  We propose Segment Anything with Robust Uncertainty-Accuracy Correlation (RUAC) for robust pixel-wise uncertainty estimation under appearance and deformation shifts.<n>RUAC adds a lightweight uncertainty head, trains it with a collaborative style-deformation attack that jointly perturbs texture and geometry.<n>Across 23 zero-shot domains, RUAC improves segmentation quality and yields more faithful uncertainty with stronger uncertainty-accuracy correlation.
  arXiv  Detail & Related papers  (2026-05-11T14:04:02Z)
• Robust Alignment: Harmonizing Clean Accuracy and Adversarial Robustness in Adversarial Training [16.873607054158793]
  Adversarial Training (AT) is one of the most effective methods for developing robust deep neural networks (DNNs)<n>We show that Varying input perturbation intensities for training samples near decision boundaries in AT have minimal impact on model robustness.<n>We propose a new Robust Alignment Adversarial Training (RAAT) method, effectively harmonizing accuracy and robustness.
  arXiv  Detail & Related papers  (2026-04-29T09:59:41Z)
• Robust 4D Visual Geometry Transformer with Uncertainty-Aware Priors [38.939519059502025]
  We present a framework designed to disentangle dynamic and static components by modeling uncertainty across different stages of the reconstruction process.<n>Our approach reduces Mean Accuracy error by 13.43% and improves segmentation F-measure by 10.49%.<n>Our framework maintains the efficiency of feed-forward inference and requires no task-specific fine-tuning or per-scene optimization.
  arXiv  Detail & Related papers  (2026-04-10T14:36:50Z)
• What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models [64.99746027349767]
  Adversarial Robustness Adaptation (R-Adapt) is a framework that freezes all pre-trained weights and introduces minimal, insight-driven adaptations only in the initial layers.<n>R-Adapt generalizes efficiently to large vision-language models (e.g., LLaVA and Qwen-VL) to enhance their robustness.
  arXiv  Detail & Related papers  (2026-03-13T09:02:11Z)
• Margin and Consistency Supervision for Calibrated and Robust Vision Models [1.0152838128195467]
  Margin and Consistency Supervision (MaCS) is a simple, architecture-agnostic regularization framework.<n>MaCS jointly enforces logit-space separation and local prediction stability.<n>We show that increasing classification margin while reducing local sensitivity formalized via a Lipschitz-type stability proxy yields improved generalization guarantees.
  arXiv  Detail & Related papers  (2026-03-06T01:54:28Z)
• Zubov-Net: Adaptive Stability for Neural ODEs Reconciling Accuracy with Robustness [0.16355471507854133]
  We propose an adaptive stable learning framework named ZubovNet, which reformulates Zubov's equation into a consistency characterization.<n>We introduce a new paradigm for actively controlling the geometry of RoAs by directly optimizing PRoAs to reconcile accuracy and robustness.
  arXiv  Detail & Related papers  (2025-09-26T05:01:51Z)
• Adaptive Dual Uncertainty Optimization: Boosting Monocular 3D Object Detection under Test-Time Shifts [80.32933059529135]
  Test-Time Adaptation (TTA) methods have emerged to adapt to target distributions during inference.<n>We propose Dual Uncertainty Optimization (DUO), the first TTA framework designed to jointly minimize both uncertainties for robust M3OD.<n>In parallel, we design a semantic-aware normal field constraint that preserves geometric coherence in regions with clear semantic cues.
  arXiv  Detail & Related papers  (2025-08-28T07:09:21Z)
• SR-Stereo & DAPE: Stepwise Regression and Pre-trained Edges for Practical Stereo Matching [2.8908326904081334]
  We propose a novel stepwise regression architecture to overcome domain discrepancies. To enhance the edge awareness of models adapting new domains with sparse ground truth, we propose Domain Adaptation based on Pre-trained Edges (DAPE) The proposed SR-Stereo and DAPE are extensively evaluated on SceneFlow, KITTI, Middbury 2014 and ETH3D.
  arXiv  Detail & Related papers  (2024-06-11T05:25:25Z)
• The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness [50.52507648690234]
  Federated learning has the risk of skewing fine-tuning features and compromising the robustness of the model. We introduce three robustness indicators and conduct experiments across diverse robust datasets. Our approach markedly enhances the robustness across diverse scenarios, encompassing various parameter-efficient fine-tuning methods.
  arXiv  Detail & Related papers  (2024-01-25T09:18:51Z)
• Towards Calibrated Robust Fine-Tuning of Vision-Language Models [97.19901765814431]
  This work proposes a robust fine-tuning method that improves both OOD accuracy and confidence calibration simultaneously in vision language models. We show that both OOD classification and OOD calibration errors have a shared upper bound consisting of two terms of ID data. Based on this insight, we design a novel framework that conducts fine-tuning with a constrained multimodal contrastive loss enforcing a larger smallest singular value.
  arXiv  Detail & Related papers  (2023-11-03T05:41:25Z)
• Robustness and Accuracy Could Be Reconcilable by (Proper) Definition [109.62614226793833]
  The trade-off between robustness and accuracy has been widely studied in the adversarial literature. We find that it may stem from the improperly defined robust error, which imposes an inductive bias of local invariance. By definition, SCORE facilitates the reconciliation between robustness and accuracy, while still handling the worst-case uncertainty.
  arXiv  Detail & Related papers  (2022-02-21T10:36:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.