SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

• URL: http://arxiv.org/abs/2603.19028v1
• Date: Thu, 19 Mar 2026 15:28:08 GMT
• Title: SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models
• Abstract summary: We propose Sparse Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder latent space.<n>SEM identifies and modulates bias-relevant neurons while preserving query-relevant ones.<n>Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of vision-language models.
• Score: 36.40093869747498
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Models that bridge vision and language, such as CLIP, are key components of multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biases. Existing post-hoc debiasing methods often operate directly in the dense CLIP embedding space, where bias and task-relevant information are highly entangled. This entanglement limits their ability to remove bias without degrading semantic fidelity. In this work, we propose Sparse Embedding Modulation (SEM), a post-hoc, zero-shot debiasing framework that operates in a Sparse Autoencoder (SAE) latent space. By decomposing CLIP text embeddings into disentangled features, SEM identifies and modulates bias-relevant neurons while preserving query-relevant ones. This enables more precise, non-linear interventions. Across four benchmark datasets and two CLIP backbones, SEM achieves substantial fairness gains in retrieval and zero-shot classification. Our results demonstrate that sparse latent representations provide an effective foundation for post-hoc debiasing of vision-language models.

Related papers

• S$^2$COPE: Self-Supervised Concept Discovery via Preference Learning [15.951097715837799]
  We introduce Self-Supervised Concept discOvery via Preference lEarning (model), a label-free framework that resolves this dilemma.<n>Our work suggest that interpretability can emerge through a model's autonomous interaction with incidental visual structures, without any human supervision.
  arXiv  Detail & Related papers  (2026-06-12T16:02:44Z)
• When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models [4.309108879640932]
  Vision-Language Models (VLMs) increasingly power high-stakes applications.<n>They routinely hallucinate, confidently describing content not present in the input.<n>We trace these failure modes to a geometric over-alignment.<n>We propose two complementary remedies: a training-free inference strategy and a bias-aware fine-tuning paradigm.
  arXiv  Detail & Related papers  (2026-05-07T10:09:18Z)
• SAGE: Spuriousness-Aware Guided Prompt Exploration for Mitigating Multimodal Bias [43.06550296269926]
  Large vision-language models, such as CLIP, have shown strong zero-shot classification performance by aligning images and text in a shared embedding space.<n>However, CLIP models often develop multimodal spurious biases, which is the undesirable tendency to rely on spurious features.<n>We propose Spuriousness-Aware Guided Exploration (SAGE), a simple and effective method that mitigates spurious bias through guided prompt selection.
  arXiv  Detail & Related papers  (2025-11-17T05:52:32Z)
• Enhancing CLIP Robustness via Cross-Modality Alignment [54.01929554563447]
  We propose Cross-modality Alignment, an optimal transport-based framework for vision-language models.<n> COLA restores global image-text alignment and local structural consistency in the feature space.<n> COLA is training-free and compatible with existing fine-tuned models.
  arXiv  Detail & Related papers  (2025-10-28T03:47:44Z)
• SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models [74.40683913645731]
  Zero-shot multi-label recognition (MLR) with Vision-Language Models (VLMs) faces significant challenges without training data, model tuning, or architectural modifications.<n>Our work proposes a novel solution treating VLMs as black boxes, leveraging scores without training data or ground truth.<n>Analysis of these prompt scores reveals VLM biases and AND''/OR' signal ambiguities, notably that maximum scores are surprisingly suboptimal compared to second-highest scores.
  arXiv  Detail & Related papers  (2025-02-24T07:15:05Z)
• Data Matters Most: Auditing Social Bias in Contrastive Vision Language Models [28.944990804599893]
  We disentangle three design factors -- model size, training-data scale, and training-data source -- by comparing CLIP and OpenCLIP.<n>We also evaluate three post-hoc, test-time debiasing strategies -- Bias Prompts, Prompt Array, and SANER.
  arXiv  Detail & Related papers  (2025-01-22T21:08:30Z)
• Refining Skewed Perceptions in Vision-Language Contrastive Models through Visual Representations [0.033483662989441935]
  Large vision-language contrastive models (VLCMs) have become foundational, demonstrating remarkable success across a variety of downstream tasks.<n>Despite their advantages, these models inherit biases from the disproportionate distribution of real-world data, leading to misconceptions about the actual environment.<n>This study presents an investigation into how a simple linear probe can effectively distill task-specific core features from CLIP's embedding for downstream applications.
  arXiv  Detail & Related papers  (2024-05-22T22:03:11Z)
• Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment [53.2701026843921]
  Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. We propose the Self Structural Semantic Alignment (S3A) framework, which extracts structural semantic information from unlabeled data while simultaneously self-learning.
  arXiv  Detail & Related papers  (2023-08-24T17:56:46Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• CLIPood: Generalizing CLIP to Out-of-Distributions [73.86353105017076]
  Contrastive language-image pre-training (CLIP) models have shown impressive zero-shot ability, but the further adaptation of CLIP on downstream tasks undesirably degrades OOD performances. We propose CLIPood, a fine-tuning method that can adapt CLIP models to OOD situations where both domain shifts and open classes may occur on unseen test data. Experiments on diverse datasets with different OOD scenarios show that CLIPood consistently outperforms existing generalization techniques.
  arXiv  Detail & Related papers  (2023-02-02T04:27:54Z)
• Debiasing Vision-Language Models via Biased Prompts [79.04467131711775]
  We propose a general approach for debiasing vision-language foundation models by projecting out biased directions in the text embedding. We show that debiasing only the text embedding with a calibrated projection matrix suffices to yield robust classifiers and fair generative models.
  arXiv  Detail & Related papers  (2023-01-31T20:09:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.