SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP

• URL: http://arxiv.org/abs/2408.10202v4
• Date: Wed, 21 May 2025 07:16:20 GMT
• Title: SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP
• Abstract summary: Large-scale vision-language models, such as CLIP, are known to contain societal bias regarding protected attributes.<n>We introduce a simple-yet-effective debiasing method called SANER that eliminates attribute information from CLIP text features only of attribute-neutral descriptions.
• Score: 41.62536201065971
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale vision-language models, such as CLIP, are known to contain societal bias regarding protected attributes (e.g., gender, age). This paper aims to address the problems of societal bias in CLIP. Although previous studies have proposed to debias societal bias through adversarial learning or test-time projecting, our comprehensive study of these works identifies two critical limitations: 1) loss of attribute information when it is explicitly disclosed in the input and 2) use of the attribute annotations during debiasing process. To mitigate societal bias in CLIP and overcome these limitations simultaneously, we introduce a simple-yet-effective debiasing method called SANER (societal attribute neutralizer) that eliminates attribute information from CLIP text features only of attribute-neutral descriptions. Experimental results show that SANER, which does not require attribute annotations and preserves original information for attribute-specific descriptions, demonstrates superior debiasing ability than the existing methods.

Related papers

• Debiasing Without Protected Attributes: Latent Concept Erasure from Textual Profiles [48.07449275130565]
  Most fairness research in NLP assumes direct access to protected attributes such as gender, race, or nationality.<n>This raises a key question: can debiasing succeed without direct access to sensitive attributes?<n>We propose H-SAL, which performs post-hoc concept and attribute erasure using self-description text as an implicit debiasing signal.
  arXiv  Detail & Related papers  (2026-06-10T13:49:27Z)
• LATex: Leveraging Attribute-based Text Knowledge for Aerial-Ground Person Re-Identification [63.07563443280147]
  We propose a novel framework named LATex for AG-ReID. It adopts prompt-tuning strategies to leverage attribute-based text knowledge. Our framework can fully leverage attribute-based text knowledge to improve the AG-ReID.
  arXiv  Detail & Related papers  (2025-03-31T04:47:05Z)
• Compositional Caching for Training-free Open-vocabulary Attribute Detection [65.46250297408974]
  We present Compositional Caching (ComCa), a training-free method for open-vocabulary attribute detection.<n>ComCa requires only the list of target attributes and objects as input, using them to populate an auxiliary cache of images.<n>Experiments on public datasets demonstrate that ComCa significantly outperforms zero-shot and cache-based baselines.
  arXiv  Detail & Related papers  (2025-03-24T21:00:37Z)
• Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition [8.950906917573986]
  Few-shot adaptation for Vision-Language Models (VLMs) presents a dilemma: balancing in-distribution accuracy with out-of-distribution generalization. Recent research has utilized low-level concepts such as visual attributes to enhance generalization. This study reveals that VLMs overly rely on a small subset of attributes on decision-making, which co-occur with the category but are not inherently part of it, spuriously correlated attributes.
  arXiv  Detail & Related papers  (2025-02-19T12:05:33Z)
• Editable Fairness: Fine-Grained Bias Mitigation in Language Models [52.66450426729818]
  We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases. FAST surpasses state-of-the-art baselines with superior debiasing performance. This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
  arXiv  Detail & Related papers  (2024-08-07T17:14:58Z)
• Practical Bias Mitigation through Proxy Sensitive Attribute Label Generation [0.0]
  We propose a two-stage approach of unsupervised embedding generation followed by clustering to obtain proxy-sensitive labels. The efficacy of our work relies on the assumption that bias propagates through non-sensitive attributes that are correlated to the sensitive attributes. Experimental results demonstrate that bias mitigation using existing algorithms such as Fair Mixup and Adversarial Debiasing yields comparable results on derived proxy labels.
  arXiv  Detail & Related papers  (2023-12-26T10:54:15Z)
• SABAF: Removing Strong Attribute Bias from Neural Networks with Adversarial Filtering [20.7209867191915]
  We propose a new method for removing attribute bias in neural networks. The proposed method achieves state-of-the-art performance in both strong and moderate bias settings.
  arXiv  Detail & Related papers  (2023-11-13T08:13:55Z)
• Information-Theoretic Bounds on The Removal of Attribute-Specific Bias From Neural Networks [20.7209867191915]
  We show that existing attribute bias removal methods are effective only when the inherent bias in the dataset is relatively weak. Our findings show that existing attribute bias removal methods are effective only when the inherent bias in the dataset is relatively weak.
  arXiv  Detail & Related papers  (2023-10-08T00:39:11Z)
• Hierarchical Visual Primitive Experts for Compositional Zero-Shot Learning [52.506434446439776]
  Compositional zero-shot learning (CZSL) aims to recognize compositions with prior knowledge of known primitives (attribute and object) We propose a simple and scalable framework called Composition Transformer (CoT) to address these issues. Our method achieves SoTA performance on several benchmarks, including MIT-States, C-GQA, and VAW-CZSL.
  arXiv  Detail & Related papers  (2023-08-08T03:24:21Z)
• Causal Effect Regularization: Automated Detection and Removal of Spurious Attributes [13.852987916253685]
  In many classification datasets, the task labels are spuriously correlated with some input attributes. We propose a method to automatically identify spurious attributes by estimating their causal effect on the label. Our method mitigates the reliance on spurious attributes even under noisy estimation of causal effects.
  arXiv  Detail & Related papers  (2023-06-19T17:17:42Z)
• Exploiting Semantic Attributes for Transductive Zero-Shot Learning [97.61371730534258]
  Zero-shot learning aims to recognize unseen classes by generalizing the relation between visual features and semantic attributes learned from the seen classes. We present a novel transductive ZSL method that produces semantic attributes of the unseen data and imposes them on the generative process. Experiments on five standard benchmarks show that our method yields state-of-the-art results for zero-shot learning.
  arXiv  Detail & Related papers  (2023-03-17T09:09:48Z)
• Semi-FairVAE: Semi-supervised Fair Representation Learning with Adversarial Variational Autoencoder [92.67156911466397]
  We propose a semi-supervised fair representation learning approach based on adversarial variational autoencoder. We use a bias-aware model to capture inherent bias information on sensitive attribute. We also use a bias-free model to learn debiased fair representations by using adversarial learning to remove bias information from them.
  arXiv  Detail & Related papers  (2022-04-01T15:57:47Z)
• Marked Attribute Bias in Natural Language Inference [0.0]
  We present a new observation of gender bias in a downstream NLP application: marked attribute bias in natural language inference. Bias in downstream applications can stem from training data, word embeddings, or be amplified by the model in use. Here we seek to understand how the intrinsic properties of word embeddings contribute to this observed marked attribute effect.
  arXiv  Detail & Related papers  (2021-09-28T20:45:02Z)
• Fairness via Representation Neutralization [60.90373932844308]
  We propose a new mitigation technique, namely, Representation Neutralization for Fairness (RNF) RNF achieves that fairness by debiasing only the task-specific classification head of DNN models. Experimental results over several benchmark datasets demonstrate our RNF framework to effectively reduce discrimination of DNN models.
  arXiv  Detail & Related papers  (2021-06-23T22:26:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.