Social perception of faces in a vision-language model

• URL: http://arxiv.org/abs/2408.14435v1
• Date: Mon, 26 Aug 2024 17:21:54 GMT
• Title: Social perception of faces in a vision-language model
• Authors: Carina I. Hausladen, Manuel Knott, Colin F. Camerer, Pietro Perona,
• Abstract summary: We explore social perception of human faces in CLIP, a widely used open-source vision-language model. We find that age, gender, and race do systematically impact CLIP's social perception of faces. We find a strong pattern of bias concerning the faces of Black women.
• Score: 11.933952003478172
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We explore social perception of human faces in CLIP, a widely used open-source vision-language model. To this end, we compare the similarity in CLIP embeddings between different textual prompts and a set of face images. Our textual prompts are constructed from well-validated social psychology terms denoting social perception. The face images are synthetic and are systematically and independently varied along six dimensions: the legally protected attributes of age, gender, and race, as well as facial expression, lighting, and pose. Independently and systematically manipulating face attributes allows us to study the effect of each on social perception and avoids confounds that can occur in wild-collected data due to uncontrolled systematic correlations between attributes. Thus, our findings are experimental rather than observational. Our main findings are three. First, while CLIP is trained on the widest variety of images and texts, it is able to make fine-grained human-like social judgments on face images. Second, age, gender, and race do systematically impact CLIP's social perception of faces, suggesting an undesirable bias in CLIP vis-a-vis legally protected attributes. Most strikingly, we find a strong pattern of bias concerning the faces of Black women, where CLIP produces extreme values of social perception across different ages and facial expressions. Third, facial expression impacts social perception more than age and lighting as much as age. The last finding predicts that studies that do not control for unprotected visual attributes may reach the wrong conclusions on bias. Our novel method of investigation, which is founded on the social psychology literature and on the experiments involving the manipulation of individual attributes, yields sharper and more reliable observations than previous observational methods and may be applied to study biases in any vision-language model.

Related papers

• Dataset Scale and Societal Consistency Mediate Facial Impression Bias in Vision-Language AI [17.101569078791492]
  We study 43 CLIP vision-language models to determine whether they learn human-like facial impression biases. We show for the first time that the the degree to which a bias is shared across a society predicts the degree to which it is reflected in a CLIP model.
  arXiv  Detail & Related papers  (2024-08-04T08:26:58Z)
• Robustness Disparities in Face Detection [64.71318433419636]
  We present the first of its kind detailed benchmark of face detection systems, specifically examining the robustness to noise of commercial and academic models. Across all the datasets and systems, we generally find that photos of individuals who are $textitmasculine presenting$, of $textitolder$, of $textitdarker skin type$, or have $textitdim lighting$ are more susceptible to errors than their counterparts in other identities.
  arXiv  Detail & Related papers  (2022-11-29T05:22:47Z)
• Explaining Bias in Deep Face Recognition via Image Characteristics [9.569575076277523]
  We evaluate ten state-of-the-art face recognition models, comparing their fairness in terms of security and usability on two data sets. We then analyze the impact of image characteristics on models performance.
  arXiv  Detail & Related papers  (2022-08-23T17:18:23Z)
• CIAO! A Contrastive Adaptation Mechanism for Non-Universal Facial Expression Recognition [80.07590100872548]
  We propose Contrastive Inhibitory Adaptati On (CIAO), a mechanism that adapts the last layer of facial encoders to depict specific affective characteristics on different datasets. CIAO presents an improvement in facial expression recognition performance over six different datasets with very unique affective representations.
  arXiv  Detail & Related papers  (2022-08-10T15:46:05Z)
• Are Commercial Face Detection Models as Biased as Academic Models? [64.71318433419636]
  We compare academic and commercial face detection systems, specifically examining robustness to noise. We find that state-of-the-art academic face detection models exhibit demographic disparities in their noise robustness. We conclude that commercial models are always as biased or more biased than an academic model.
  arXiv  Detail & Related papers  (2022-01-25T02:21:42Z)
• Measuring Hidden Bias within Face Recognition via Racial Phenotypes [21.74534280021516]
  This study introduces an alternative racial bias analysis methodology via facial phenotype attributes for face recognition. We propose categorical test cases to investigate the individual influence of those attributes on bias within face recognition tasks.
  arXiv  Detail & Related papers  (2021-10-19T10:46:59Z)
• I Only Have Eyes for You: The Impact of Masks On Convolutional-Based Facial Expression Recognition [78.07239208222599]
  We evaluate how the recently proposed FaceChannel adapts towards recognizing facial expressions from persons with masks. We also perform specific feature-level visualization to demonstrate how the inherent capabilities of the FaceChannel to learn and combine facial features change when in a constrained social interaction scenario.
  arXiv  Detail & Related papers  (2021-04-16T20:03:30Z)
• Facial Expressions as a Vulnerability in Face Recognition [73.85525896663371]
  This work explores facial expression bias as a security vulnerability of face recognition systems. We present a comprehensive analysis of how facial expression bias impacts the performance of face recognition technologies.
  arXiv  Detail & Related papers  (2020-11-17T18:12:41Z)
• Learning Emotional-Blinded Face Representations [77.7653702071127]
  We propose two face representations that are blind to facial expressions associated to emotional responses. This work is motivated by new international regulations for personal data protection.
  arXiv  Detail & Related papers  (2020-09-18T09:24:10Z)
• A Set of Distinct Facial Traits Learned by Machines Is Not Predictive of Appearance Bias in the Wild [3.0349733976070015]
  We train a learning model on human subjects' first impressions of personality traits in other faces as measured by social psychologists. We find that features extracted with FaceNet can be used to predict human appearance bias scores for deliberately manipulated faces. In contrast to work with human biases in social psychology, the model does not find a significant signal correlating politicians' vote shares with perceived competence bias.
  arXiv  Detail & Related papers  (2020-02-13T17:09:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.