Beyond Performance Disparities: A Three-Level Audit of Representational Harm in CelebA

• URL: http://arxiv.org/abs/2605.15312v1
• Date: Thu, 14 May 2026 18:25:17 GMT
• Title: Beyond Performance Disparities: A Three-Level Audit of Representational Harm in CelebA
• Abstract summary: This paper examines how gendered double standards of ageing and beauty are encoded in the data and reproduced in model behaviour.<n>Female faces incur steep penalties when assigned to ageing or masculine-coded clusters.<n>Older males attain the highest accuracy but the lowest average precision, indicating categorical exclusion of groups outside the dataset's evaluative templates.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale facial datasets like CelebA are widely used in computer vision, yet the cultural biases embedded in their labels remain underexplored. Fairness research has distinguished representational from allocational harms, but audits of computer vision datasets have mostly examined categorical labels, leaving open how such harms appear in learned features and model attention. This paper examines CelebA at three levels: dataset structure, learned feature weights, and spatial attention, focusing on how gendered double standards of ageing and beauty are encoded in the data and reproduced in model behaviour. First, hierarchical clustering of 202,599 images shows that the 39 attributes organise into latent trait bundles aligned with cultural archetypes: performative femininity (youth, makeup, adornment) and professional masculinity (ageing, facial hair, formal attire). Female faces, though more often rated attractive overall, incur steep penalties when assigned to ageing or masculine-coded clusters. Second, XGBoost with SHAP analysis reveal gender-specific effects, such as adiposity reducing attractiveness only for females. Third, Grad-CAM finds that predictions for female and younger male subgroups concentrate on mid-face cues, whereas predictions for older males drift toward peripheral cues such as hair and clothing. Older males attain the highest accuracy but the lowest average precision, indicating categorical exclusion of groups outside the dataset's evaluative templates. Cultural double standards thus pass from media representation into dataset labels, feature weights, and model attention, producing two representational harms: hyper-scrutiny of women under a narrow evaluative template, and exclusion of older men from the scheme entirely. Fairness metrics focused on performance disparities mask both, underscoring the need to address representational harm in fairness research.

Related papers

• Auditing and Mitigating Bias in Gender Classification Algorithms: A Data-Centric Approach [0.0]
  We audit five widely used gender classification datasets, revealing significant intersectional underrepresentation.<n>We train identical MobileNetV2 classifiers on the two most balanced of these datasets, UTKFace and FairFace.<n>Our fairness evaluation shows that even these models exhibit significant bias, misclassifying female faces at a higher rate than male faces.
  arXiv  Detail & Related papers  (2025-10-17T02:09:17Z)
• African Gender Classification Using Clothing Identification Via Deep Learning [0.0]
  We use the AFRIFASHION1600 dataset, a curated collection of 1,600 images of African traditional clothing labeled into two gender classes: male and female.<n>A deep learning model, based on a modified VGG16 architecture and trained using transfer learning, was developed for classification.<n>The model achieved an accuracy of 87% on the test set, demonstrating strong predictive capability despite dataset imbalances favoring female samples.
  arXiv  Detail & Related papers  (2025-02-26T20:59:59Z)
• On the "Illusion" of Gender Bias in Face Recognition: Explaining the Fairness Issue Through Non-demographic Attributes [7.602456562464879]
  Face recognition systems exhibit significant accuracy differences based on the user's gender.<n>We propose a toolchain to effectively decorrelate and aggregate facial attributes to enable a less-biased gender analysis.<n>Experiments show that the gender gap vanishes when images of male and female subjects share specific attributes.
  arXiv  Detail & Related papers  (2025-01-21T10:21:19Z)
• GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing [72.0343083866144]
  This paper introduces the GenderBias-emphVL benchmark to evaluate occupation-related gender bias in Large Vision-Language Models. Using our benchmark, we extensively evaluate 15 commonly used open-source LVLMs and state-of-the-art commercial APIs. Our findings reveal widespread gender biases in existing LVLMs.
  arXiv  Detail & Related papers  (2024-06-30T05:55:15Z)
• The Male CEO and the Female Assistant: Evaluation and Mitigation of Gender Biases in Text-To-Image Generation of Dual Subjects [58.27353205269664]
  We propose the Paired Stereotype Test (PST) framework, which queries T2I models to depict two individuals assigned with male-stereotyped and female-stereotyped social identities.<n>PST queries T2I models to depict two individuals assigned with male-stereotyped and female-stereotyped social identities.<n>Using PST, we evaluate two aspects of gender biases -- the well-known bias in gendered occupation and a novel aspect: bias in organizational power.
  arXiv  Detail & Related papers  (2024-02-16T21:32:27Z)
• VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution [80.57383975987676]
  VisoGender is a novel dataset for benchmarking gender bias in vision-language models. We focus on occupation-related biases within a hegemonic system of binary gender, inspired by Winograd and Winogender schemas. We benchmark several state-of-the-art vision-language models and find that they demonstrate bias in resolving binary gender in complex scenes.
  arXiv  Detail & Related papers  (2023-06-21T17:59:51Z)
• Gender Biases in Automatic Evaluation Metrics for Image Captioning [87.15170977240643]
  We conduct a systematic study of gender biases in model-based evaluation metrics for image captioning tasks. We demonstrate the negative consequences of using these biased metrics, including the inability to differentiate between biased and unbiased generations. We present a simple and effective way to mitigate the metric bias without hurting the correlations with human judgments.
  arXiv  Detail & Related papers  (2023-05-24T04:27:40Z)
• Gender Stereotyping Impact in Facial Expression Recognition [1.5340540198612824]
  In recent years, machine learning-based models have become the most popular approach to Facial Expression Recognition (FER) In publicly available FER datasets, apparent gender representation is usually mostly balanced, but their representation in the individual label is not. We generate derivative datasets with different amounts of stereotypical bias by altering the gender proportions of certain labels. We observe a discrepancy in the recognition of certain emotions between genders of up to $29 %$ under the worst bias conditions.
  arXiv  Detail & Related papers  (2022-10-11T10:52:23Z)
• Are Commercial Face Detection Models as Biased as Academic Models? [64.71318433419636]
  We compare academic and commercial face detection systems, specifically examining robustness to noise. We find that state-of-the-art academic face detection models exhibit demographic disparities in their noise robustness. We conclude that commercial models are always as biased or more biased than an academic model.
  arXiv  Detail & Related papers  (2022-01-25T02:21:42Z)
• Mitigating Gender Bias in Captioning Systems [56.25457065032423]
  Most captioning models learn gender bias, leading to high gender prediction errors, especially for women. We propose a new Guided Attention Image Captioning model (GAIC) which provides self-guidance on visual attention to encourage the model to capture correct gender visual evidence.
  arXiv  Detail & Related papers  (2020-06-15T12:16:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.