Birds of a Feather Flock Together: Background-Invariant Representations via Linear Structure in VLMs

• URL: http://arxiv.org/abs/2605.11107v1
• Date: Mon, 11 May 2026 18:13:05 GMT
• Title: Birds of a Feather Flock Together: Background-Invariant Representations via Linear Structure in VLMs
• Abstract summary: We show that correlations between foreground objects and their backgrounds constitute a salient and practically important class of spurious dependencies.<n>We introduce a pre-training approach that exploits this property to construct background-invariant representations using synthetic data.<n>Our method achieves, to our knowledge, the first worst-group accuracy exceeding $90%$ on Waterbirds under perfect ($100%$) spurious correlation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs), such as CLIP and SigLIP 2, are widely used for image classification, yet their vision encoders remain vulnerable to systematic biases that undermine robustness. In particular, correlations between foreground objects and their backgrounds constitute a salient and practically important class of spurious dependencies. In this work, we revisit the well-known property of high linear additivity in VLM embedding spaces and show that it enables a decomposition of scene representations into foreground and background components. Leveraging this insight, we introduce a pre-training approach that exploits this property to construct background-invariant representations using synthetic data. Our method achieves, to our knowledge, the first worst-group accuracy exceeding $90\%$ on Waterbirds under perfect ($100\%$) spurious correlation (i.e., no minority-group examples in the training data). Furthermore, it demonstrates strong sim-to-real transfer and requires no access to real-world debiased data, making it practical for real-world deployment.

Related papers

• Dual-Foundation Models for Unsupervised Domain Adaptation [2.279449016085348]
  Training segmentation models requires costly, labor-intensive annotations on real-world datasets.<n>Unsupervised Domain Adaptation (UDA) addresses this by training models on labeled synthetic data and adapting them to unlabeled real images.<n>We propose a dual-foundation UDA framework that leverages two complementary foundation models.
  arXiv  Detail & Related papers  (2026-05-05T04:52:08Z)
• Fine-tuning MLLMs Without Forgetting Is Easier Than You Think [72.59321247529975]
  We design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs.<n>Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images.<n>We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy.
  arXiv  Detail & Related papers  (2026-03-15T17:16:19Z)
• Toward Generalizable Deblurring: Leveraging Massive Blur Priors with Linear Attention for Real-World Scenarios [9.82847623835017]
  GLOWDeblur is a Generalizable reaL-wOrld lightWeight Deblur model that combines convolution-based pre-reconstruction & domain alignment module with a lightweight diffusion backbone.<n>We propose Blur Pattern Pretraining (BPP), which acquires blur priors from simulation datasets and transfers them through joint fine-tuning on real data.<n>We further introduce Motion and Semantic Guidance (MoSeG) to strengthen blur priors under severe degradation, and integrate it into GLOWDeblur, a Generalizable reaL-wOrld lightWeight Deblur model that combines convolution-based pre-reconstruction &
  arXiv  Detail & Related papers  (2026-01-10T11:01:31Z)
• Mono2Stereo: Monocular Knowledge Transfer for Enhanced Stereo Matching [7.840781070208874]
  We propose leveraging monocular knowledge transfer to enhance stereo matching, namely Mono2Stereo. We introduce knowledge transfer with a two-stage training process, comprising synthetic data pre-training and real-world data fine-tuning. Experimental results demonstrate that our pre-trained model exhibits strong zero-shot capabilities.
  arXiv  Detail & Related papers  (2024-11-14T03:01:36Z)
• Refining Skewed Perceptions in Vision-Language Contrastive Models through Visual Representations [0.033483662989441935]
  Large vision-language contrastive models (VLCMs) have become foundational, demonstrating remarkable success across a variety of downstream tasks.<n>Despite their advantages, these models inherit biases from the disproportionate distribution of real-world data, leading to misconceptions about the actual environment.<n>This study presents an investigation into how a simple linear probe can effectively distill task-specific core features from CLIP's embedding for downstream applications.
  arXiv  Detail & Related papers  (2024-05-22T22:03:11Z)
• Fake It Till Make It: Federated Learning with Consensus-Oriented Generation [52.82176415223988]
  We propose federated learning with consensus-oriented generation (FedCOG) FedCOG consists of two key components at the client side: complementary data generation and knowledge-distillation-based model training. Experiments on classical and real-world FL datasets show that FedCOG consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-10T18:49:59Z)
• Mitigating the Effect of Incidental Correlations on Part-based Learning [50.682498099720114]
  Part-based representations could be more interpretable and generalize better with limited data. We present two innovative regularization methods for part-based representations. We exhibit state-of-the-art (SoTA) performance on few-shot learning tasks on benchmark datasets.
  arXiv  Detail & Related papers  (2023-09-30T13:44:48Z)
• ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
  We introduce Action-Aware Embodied Learning for Perception (ALP) ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. We show that ALP outperforms existing baselines in several downstream perception tasks.
  arXiv  Detail & Related papers  (2023-06-16T21:51:04Z)
• Imposing Consistency for Optical Flow Estimation [73.53204596544472]
  Imposing consistency through proxy tasks has been shown to enhance data-driven learning. This paper introduces novel and effective consistency strategies for optical flow estimation.
  arXiv  Detail & Related papers  (2022-04-14T22:58:30Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)
• Transductive Zero-Shot Learning by Decoupled Feature Generation [30.664199050468472]
  We focus on the transductive setting, in which unlabelled visual data from unseen classes is available. We propose to decouple tasks of generating realistic visual features and translating semantic attributes into visual cues. We present a detailed ablation study to dissect the effect of our proposed decoupling approach, while demonstrating its superiority over the related state-of-the-art.
  arXiv  Detail & Related papers  (2021-02-05T16:17:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.