TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tone

• URL: http://arxiv.org/abs/2505.20637v1
• Date: Tue, 27 May 2025 02:31:08 GMT
• Title: TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tone
• Authors: Ana M. Cabanas, Alma Pedro, Domingo Mery,
• Abstract summary: This study compares two objective skin tone classification methods: the widely used Individual Typology Angle (ITA) and a perceptually grounded alternative based on Lightness ($L*$) and Hue ($H*$)<n>Using AffectNet and a MobileNet-based model, we assess fairness across skin tone groups defined by each method.
• Score: 4.847470451539328
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding how facial affect analysis (FAA) systems perform across different demographic groups requires reliable measurement of sensitive attributes such as ancestry, often approximated by skin tone, which itself is highly influenced by lighting conditions. This study compares two objective skin tone classification methods: the widely used Individual Typology Angle (ITA) and a perceptually grounded alternative based on Lightness ($L^*$) and Hue ($H^*$). Using AffectNet and a MobileNet-based model, we assess fairness across skin tone groups defined by each method. Results reveal a severe underrepresentation of dark skin tones ($\sim 2 \%$), alongside fairness disparities in F1-score (up to 0.08) and TPR (up to 0.11) across groups. While ITA shows limitations due to its sensitivity to lighting, the $H^*$-$L^*$ method yields more consistent subgrouping and enables clearer diagnostics through metrics such as Equal Opportunity. Grad-CAM analysis further highlights differences in model attention patterns by skin tone, suggesting variation in feature encoding. To support future mitigation efforts, we also propose a modular fairness-aware pipeline that integrates perceptual skin tone estimation, model interpretability, and fairness evaluation. These findings emphasize the relevance of skin tone measurement choices in fairness assessment and suggest that ITA-based evaluations may overlook disparities affecting darker-skinned individuals.

Related papers

• Skin Color Measurement from Dermatoscopic Images: An Evaluation on a Synthetic Dataset [0.0]
  We assess four classes of image colorimetry approaches: segmentation-based, patch-based, color quantization, and neural networks.<n>Our results show that segmentation-based and color quantization methods yield robust, lighting-invariant estimates.<n>Neural network models, particularly when combined with heavy blurring to reduce overfitting, can provide light-invariant Fitzpatrick predictions.
  arXiv  Detail & Related papers  (2025-04-06T13:57:34Z)
• FairSkin: Fair Diffusion for Skin Disease Image Generation [54.29840149709033]
  Diffusion Model (DM) has become a leading method in generating synthetic medical images, but it suffers from a critical twofold bias. We propose FairSkin, a novel DM framework that mitigates these biases through a three-level resampling mechanism. Our approach significantly improves the diversity and quality of generated images, contributing to more equitable skin disease detection in clinical settings.
  arXiv  Detail & Related papers  (2024-10-29T21:37:03Z)
• Evaluating Machine Learning-based Skin Cancer Diagnosis [0.0]
  The research assesses two convolutional neural network architectures: a MobileNet-based model and a custom CNN model. Both models are evaluated for their ability to classify skin lesions into seven categories and to distinguish between dangerous and benign lesions. The study concludes that while the models show promise in explainability, further development is needed to ensure fairness across different skin tones.
  arXiv  Detail & Related papers  (2024-09-04T02:44:48Z)
• DDI-CoCo: A Dataset For Understanding The Effect Of Color Contrast In Machine-Assisted Skin Disease Detection [51.92255321684027]
  We study the interaction between skin tone and color difference effects and suggest that color difference can be an additional reason behind model performance bias between skin tones. Our work provides a complementary angle to dermatology AI for improving skin disease detection.
  arXiv  Detail & Related papers  (2024-01-24T07:45:24Z)
• Evaluating the Fairness of Discriminative Foundation Models in Computer Vision [51.176061115977774]
  We propose a novel taxonomy for bias evaluation of discriminative foundation models, such as Contrastive Language-Pretraining (CLIP) We then systematically evaluate existing methods for mitigating bias in these models with respect to our taxonomy. Specifically, we evaluate OpenAI's CLIP and OpenCLIP models for key applications, such as zero-shot classification, image retrieval and image captioning.
  arXiv  Detail & Related papers  (2023-10-18T10:32:39Z)
• Revisiting Skin Tone Fairness in Dermatological Lesion Classification [3.247628857305427]
  We review and compare four ITA-based approaches of skin tone classification on the ISIC18 dataset. Our analyses reveal a high disagreement among previously published studies demonstrating the risks of ITA-based skin tone estimation methods. We investigate the causes of such large discrepancy among these approaches and find that the lack of diversity in the ISIC18 dataset limits its use as a testbed for fairness analysis.
  arXiv  Detail & Related papers  (2023-08-18T15:59:55Z)
• FairDisCo: Fairer AI in Dermatology via Disentanglement Contrastive Learning [11.883809920936619]
  We propose FairDisCo, a disentanglement deep learning framework with contrastive learning. We compare FairDisCo to three fairness methods, namely, resampling, reweighting, and attribute-aware. We adapt two fairness-based metrics DPM and EOM for our multiple classes and sensitive attributes task, highlighting the skin-type bias in skin lesion classification.
  arXiv  Detail & Related papers  (2022-08-22T01:54:23Z)
• Meta Balanced Network for Fair Face Recognition [51.813457201437195]
  We systematically and scientifically study bias from both data and algorithm aspects. We propose a novel meta-learning algorithm, called Meta Balanced Network (MBN), which learns adaptive margins in large margin loss. Extensive experiments show that MBN successfully mitigates bias and learns more balanced performance for people with different skin tones in face recognition.
  arXiv  Detail & Related papers  (2022-05-13T10:25:44Z)
• Automatic Facial Skin Feature Detection for Everyone [60.31670960526022]
  We present an automatic facial skin feature detection method that works across a variety of skin tones and age groups for selfies in the wild. To be specific, we annotate the locations of acne, pigmentation, and wrinkle for selfie images with different skin tone colors, severity levels, and lighting conditions.
  arXiv  Detail & Related papers  (2022-03-30T04:52:54Z)
• Reliability and Validity of Image-Based and Self-Reported Skin Phenotype Metrics [0.0]
  We show that measures of skin-tone for biometric performance evaluations must come from objective, characterized, and controlled sources. Results demonstrate that measures of skin-tone for biometric performance evaluations must come from objective, characterized, and controlled sources.
  arXiv  Detail & Related papers  (2021-06-18T16:12:24Z)
• Estimating and Improving Fairness with Adversarial Learning [65.99330614802388]
  We propose an adversarial multi-task training strategy to simultaneously mitigate and detect bias in the deep learning-based medical image analysis system. Specifically, we propose to add a discrimination module against bias and a critical module that predicts unfairness within the base classification model. We evaluate our framework on a large-scale public-available skin lesion dataset.
  arXiv  Detail & Related papers  (2021-03-07T03:10:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.