Mitigating Bias with Words: Inducing Demographic Ambiguity in Face Recognition Templates by Text Encoding

• URL: http://arxiv.org/abs/2512.08981v1
• Date: Fri, 05 Dec 2025 10:41:58 GMT
• Title: Mitigating Bias with Words: Inducing Demographic Ambiguity in Face Recognition Templates by Text Encoding
• Authors: Tahar Chettaoui, Naser Damer, Fadi Boutros,
• Abstract summary: Face recognition systems are often prone to demographic biases, partially due to the entanglement of demographic-specific information with identity-relevant features in facial embeddings.<n>We propose a novel strategy, Unified Text-Image Embedding (UTIE), which aims to induce demographic ambiguity in face embeddings.<n>We show that UTIE consistently reduces bias metrics while maintaining, or even improving in several cases, the face verification accuracy.
• Score: 18.08946802592489
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Face recognition (FR) systems are often prone to demographic biases, partially due to the entanglement of demographic-specific information with identity-relevant features in facial embeddings. This bias is extremely critical in large multicultural cities, especially where biometrics play a major role in smart city infrastructure. The entanglement can cause demographic attributes to overshadow identity cues in the embedding space, resulting in disparities in verification performance across different demographic groups. To address this issue, we propose a novel strategy, Unified Text-Image Embedding (UTIE), which aims to induce demographic ambiguity in face embeddings by enriching them with information related to other demographic groups. This encourages face embeddings to emphasize identity-relevant features and thus promotes fairer verification performance across groups. UTIE leverages the zero-shot capabilities and cross-modal semantic alignment of Vision-Language Models (VLMs). Given that VLMs are naturally trained to align visual and textual representations, we enrich the facial embeddings of each demographic group with text-derived demographic features extracted from other demographic groups. This encourages a more neutral representation in terms of demographic attributes. We evaluate UTIE using three VLMs, CLIP, OpenCLIP, and SigLIP, on two widely used benchmarks, RFW and BFW, designed to assess bias in FR. Experimental results show that UTIE consistently reduces bias metrics while maintaining, or even improving in several cases, the face verification accuracy.

Related papers

• Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos [79.03150233804458]
  Real-world images entangle race and gender with correlated factors such as background and clothing, obscuring attribution.<n>We propose a textbfface-only counterfactual evaluation paradigm<n>We generate counterfactual variants by editing only facial attributes related to race and gender, keeping all other visual factors fixed.
  arXiv  Detail & Related papers  (2026-01-11T14:35:06Z)
• DAIQ: Auditing Demographic Attribute Inference from Question in LLMs [3.1677998308405786]
  Large Language Models (LLMs) are known to reflect social biases when demographic attributes, such as gender or race, are explicitly present in the input.<n>But even in their absence, these models still infer user identities based solely on question phrasing.<n>We introduce Demographic Attribute Inference from Questions (DAIQ), a task and framework for auditing an overlooked failure mode in language models.
  arXiv  Detail & Related papers  (2025-08-18T19:26:17Z)
• Balancing the Scales: Enhancing Fairness in Facial Expression Recognition with Latent Alignment [5.784550537553534]
  This workleverages representation learning based on latent spaces to mitigate bias in facial expression recognition systems. It also enhances a deep learning model's fairness and overall accuracy.
  arXiv  Detail & Related papers  (2024-10-25T10:03:10Z)
• GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing [72.0343083866144]
  This paper introduces the GenderBias-emphVL benchmark to evaluate occupation-related gender bias in Large Vision-Language Models. Using our benchmark, we extensively evaluate 15 commonly used open-source LVLMs and state-of-the-art commercial APIs. Our findings reveal widespread gender biases in existing LVLMs.
  arXiv  Detail & Related papers  (2024-06-30T05:55:15Z)
• Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMs [13.744746481528711]
  Large Language Models (LLMs) are widely used to simulate human responses across diverse contexts.<n>We evaluate nine popular LLMs on their ability to understand demographic differences in two subjective judgment tasks: politeness and offensiveness.<n>We find that in zero-shot settings, most models' predictions for both tasks align more closely with labels from White participants than those from Asian or Black participants.
  arXiv  Detail & Related papers  (2023-11-16T10:02:24Z)
• Invariant Feature Regularization for Fair Face Recognition [45.23154294914808]
  We show that biased feature generalizes poorly in the minority group. We propose to generate diverse data partitions iteratively in an unsupervised fashion. INV-REG leads to new state-of-the-art that improves face recognition on a variety of demographic groups.
  arXiv  Detail & Related papers  (2023-10-23T07:44:12Z)
• Balancing Biases and Preserving Privacy on Balanced Faces in the Wild [50.915684171879036]
  There are demographic biases present in current facial recognition (FR) models. We introduce our Balanced Faces in the Wild dataset to measure these biases across different ethnic and gender subgroups. We find that relying on a single score threshold to differentiate between genuine and imposters sample pairs leads to suboptimal results. We propose a novel domain adaptation learning scheme that uses facial features extracted from state-of-the-art neural networks.
  arXiv  Detail & Related papers  (2021-03-16T15:05:49Z)
• Mitigating Face Recognition Bias via Group Adaptive Classifier [53.15616844833305]
  This work aims to learn a fair face representation, where faces of every group could be more equally represented. Our work is able to mitigate face recognition bias across demographic groups while maintaining the competitive accuracy.
  arXiv  Detail & Related papers  (2020-06-13T06:43:37Z)
• Enhancing Facial Data Diversity with Style-based Face Aging [59.984134070735934]
  In particular, face datasets are typically biased in terms of attributes such as gender, age, and race. We propose a novel, generative style-based architecture for data augmentation that captures fine-grained aging patterns. We show that the proposed method outperforms state-of-the-art algorithms for age transfer.
  arXiv  Detail & Related papers  (2020-06-06T21:53:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.