Skin Deep: Investigating Subjectivity in Skin Tone Annotations for Computer Vision Benchmark Datasets

• URL: http://arxiv.org/abs/2305.09072v1
• Date: Mon, 15 May 2023 23:55:56 GMT
• Title: Skin Deep: Investigating Subjectivity in Skin Tone Annotations for Computer Vision Benchmark Datasets
• Authors: Teanna Barrett, Quan Ze Chen, Amy X. Zhang
• Abstract summary: We investigate how subjective understandings of skin tone are embedded in skin tone annotation procedures. Our work is the first to investigate the skin tone annotation process as a sociotechnical project.
• Score: 7.519872646378834
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: To investigate the well-observed racial disparities in computer vision systems that analyze images of humans, researchers have turned to skin tone as more objective annotation than race metadata for fairness performance evaluations. However, the current state of skin tone annotation procedures is highly varied. For instance, researchers use a range of untested scales and skin tone categories, have unclear annotation procedures, and provide inadequate analyses of uncertainty. In addition, little attention is paid to the positionality of the humans involved in the annotation process--both designers and annotators alike--and the historical and sociological context of skin tone in the United States. Our work is the first to investigate the skin tone annotation process as a sociotechnical project. We surveyed recent skin tone annotation procedures and conducted annotation experiments to examine how subjective understandings of skin tone are embedded in skin tone annotation procedures. Our systematic literature review revealed the uninterrogated association between skin tone and race and the limited effort to analyze annotator uncertainty in current procedures for skin tone annotation in computer vision evaluation. Our experiments demonstrated that design decisions in the annotation procedure such as the order in which the skin tone scale is presented or additional context in the image (i.e., presence of a face) significantly affected the resulting inter-annotator agreement and individual uncertainty of skin tone annotations. We call for greater reflexivity in the design, analysis, and documentation of procedures for evaluation using skin tone.

Related papers

• Colorimetric skin tone scale for improved accuracy and reduced perceptual bias of human skin tone annotations [0.0]
  We develop a novel Colorimetric Skin Tone (CST) scale based on prior colorimetric measurements. Using experiments requiring humans to rate their own skin tone and the skin tone of subjects in images, we show that the new CST scale is more sensitive, consistent, and colorimetrically accurate.
  arXiv  Detail & Related papers  (2024-10-28T13:29:24Z)
• DDI-CoCo: A Dataset For Understanding The Effect Of Color Contrast In Machine-Assisted Skin Disease Detection [51.92255321684027]
  We study the interaction between skin tone and color difference effects and suggest that color difference can be an additional reason behind model performance bias between skin tones. Our work provides a complementary angle to dermatology AI for improving skin disease detection.
  arXiv  Detail & Related papers  (2024-01-24T07:45:24Z)
• Revisiting Skin Tone Fairness in Dermatological Lesion Classification [3.247628857305427]
  We review and compare four ITA-based approaches of skin tone classification on the ISIC18 dataset. Our analyses reveal a high disagreement among previously published studies demonstrating the risks of ITA-based skin tone estimation methods. We investigate the causes of such large discrepancy among these approaches and find that the lack of diversity in the ISIC18 dataset limits its use as a testbed for fairness analysis.
  arXiv  Detail & Related papers  (2023-08-18T15:59:55Z)
• ColorSense: A Study on Color Vision in Machine Visual Recognition [57.916512479603064]
  We collect 110,000 non-trivial human annotations of foreground and background color labels from visual recognition benchmarks. We validate the use of our datasets by demonstrating that the level of color discrimination has a dominating effect on the performance of machine perception models. Our findings suggest that object recognition tasks such as classification and localization are susceptible to color vision bias.
  arXiv  Detail & Related papers  (2022-12-16T18:51:41Z)
• Meta Balanced Network for Fair Face Recognition [51.813457201437195]
  We systematically and scientifically study bias from both data and algorithm aspects. We propose a novel meta-learning algorithm, called Meta Balanced Network (MBN), which learns adaptive margins in large margin loss. Extensive experiments show that MBN successfully mitigates bias and learns more balanced performance for people with different skin tones in face recognition.
  arXiv  Detail & Related papers  (2022-05-13T10:25:44Z)
• Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation [48.632358823108326]
  Virtual facial avatars will play an increasingly important role in immersive communication, games and the metaverse. Virtual facial avatars will play an increasingly important role in immersive communication, games and the metaverse. This requires accurate recovery of the appearance, represented by albedo, regardless of age, sex, or ethnicity.
  arXiv  Detail & Related papers  (2022-05-08T22:01:30Z)
• Color Invariant Skin Segmentation [17.501659517108884]
  This paper addresses the problem of automatically detecting human skin in images without reliance on color information. A primary motivation of the work has been to achieve results that are consistent across the full range of skin tones. We present a new approach that performs well in the absence of such information.
  arXiv  Detail & Related papers  (2022-04-21T05:07:21Z)
• Automatic Facial Skin Feature Detection for Everyone [60.31670960526022]
  We present an automatic facial skin feature detection method that works across a variety of skin tones and age groups for selfies in the wild. To be specific, we annotate the locations of acne, pigmentation, and wrinkle for selfie images with different skin tone colors, severity levels, and lighting conditions.
  arXiv  Detail & Related papers  (2022-03-30T04:52:54Z)
• Detecting Melanoma Fairly: Skin Tone Detection and Debiasing for Skin Lesion Classification [5.71097144710995]
  We use a modified variational autoencoder to uncover skin tone bias in datasets commonly used as benchmarks. We propose an efficient yet effective algorithm for automatically labelling the skin tone of lesion images. We subsequently use two leading bias unlearning techniques to mitigate skin tone bias.
  arXiv  Detail & Related papers  (2022-02-06T18:53:06Z)
• Anatomizing Bias in Facial Analysis [86.79402670904338]
  Existing facial analysis systems have been shown to yield biased results against certain demographic subgroups. It has become imperative to ensure that these systems do not discriminate based on gender, identity, or skin tone of individuals. This has led to research in the identification and mitigation of bias in AI systems.
  arXiv  Detail & Related papers  (2021-12-13T09:51:13Z)
• Analysis of Manual and Automated Skin Tone Assignments for Face Recognition Applications [8.334167427229572]
  We analyze a set of manual Fitzpatrick skin type assignments and also employ the individual typology angle to automatically estimate the skin tone from face images. The level of agreement between automated and manual approaches is found to be 96% or better for the MORPH images.
  arXiv  Detail & Related papers  (2021-04-29T22:35:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.