Rethinking Affect Analysis: A Protocol for Ensuring Fairness and Consistency

• URL: http://arxiv.org/abs/2408.02164v2
• Date: Wed, 7 Aug 2024 09:23:36 GMT
• Title: Rethinking Affect Analysis: A Protocol for Ensuring Fairness and Consistency
• Authors: Guanyu Hu, Dimitrios Kollias, Eleni Papadopoulou, Paraskevi Tzouveli, Jie Wei, Xinyu Yang,
• Abstract summary: We propose a unified protocol for database partitioning that ensures fairness and comparability. We provide detailed demographic annotations (in terms of race, gender and age), evaluation metrics, and a common framework for expression recognition. We also rerun the methods with the new protocol and introduce a new leaderboards to encourage future research in affect recognition with a fairer comparison.
• Score: 24.737468736951374
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Evaluating affect analysis methods presents challenges due to inconsistencies in database partitioning and evaluation protocols, leading to unfair and biased results. Previous studies claim continuous performance improvements, but our findings challenge such assertions. Using these insights, we propose a unified protocol for database partitioning that ensures fairness and comparability. We provide detailed demographic annotations (in terms of race, gender and age), evaluation metrics, and a common framework for expression recognition, action unit detection and valence-arousal estimation. We also rerun the methods with the new protocol and introduce a new leaderboards to encourage future research in affect recognition with a fairer comparison. Our annotations, code, and pre-trained models are available on \hyperlink{https://github.com/dkollias/Fair-Consistent-Affect-Analysis}{Github}.

Related papers

• Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness [16.623648447423438]
  Recent work has proposed neural network pruning techniques to reduce the size of a network while preserving robustness against adversarial examples. These methods involve complex and articulated designs, making it difficult to analyze the differences and establish a fair and accurate comparison. We propose a novel taxonomy to categorize them based on two main dimensions: the pipeline, defining when to prune; and the specifics, defining how to prune.
  arXiv  Detail & Related papers  (2024-09-02T13:34:01Z)
• Bridging the Gap: Protocol Towards Fair and Consistent Affect Analysis [24.737468736951374]
  The increasing integration of machine learning algorithms in daily life underscores the critical need for fairness and equity in their deployment. Existing databases and methodologies lack uniformity, leading to biased evaluations. This work addresses these issues by analyzing six affective databases, annotating demographic attributes, and proposing a common protocol for database partitioning.
  arXiv  Detail & Related papers  (2024-05-10T22:40:01Z)
• Backdoor-based Explainable AI Benchmark for High Fidelity Evaluation of Attribution Methods [49.62131719441252]
  Attribution methods compute importance scores for input features to explain the output predictions of deep models. In this work, we first identify a set of fidelity criteria that reliable benchmarks for attribution methods are expected to fulfill. We then introduce a Backdoor-based eXplainable AI benchmark (BackX) that adheres to the desired fidelity criteria.
  arXiv  Detail & Related papers  (2024-05-02T13:48:37Z)
• Cobra Effect in Reference-Free Image Captioning Metrics [58.438648377314436]
  A proliferation of reference-free methods, leveraging visual-language pre-trained models (VLMs), has emerged. In this paper, we study if there are any deficiencies in reference-free metrics. We employ GPT-4V as an evaluative tool to assess generated sentences and the result reveals that our approach achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2024-02-18T12:36:23Z)
• Fairness meets Cross-Domain Learning: a new perspective on Models and Metrics [80.07271410743806]
  We study the relationship between cross-domain learning (CD) and model fairness. We introduce a benchmark on face and medical images spanning several demographic groups as well as classification and localization tasks. Our study covers 14 CD approaches alongside three state-of-the-art fairness algorithms and shows how the former can outperform the latter.
  arXiv  Detail & Related papers  (2023-03-25T09:34:05Z)
• TRUE: Re-evaluating Factual Consistency Evaluation [29.888885917330327]
  We introduce TRUE: a comprehensive study of factual consistency metrics on a standardized collection of existing texts from diverse tasks. Our standardization enables an example-level meta-evaluation protocol that is more actionable and interpretable than previously reported correlations. Across diverse state-of-the-art metrics and 11 datasets we find that large-scale NLI and question generation-and-answering-based approaches achieve strong and complementary results.
  arXiv  Detail & Related papers  (2022-04-11T10:14:35Z)
• FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment [93.09267863425492]
  We argue that understanding both high-level semantics and internal temporal structures of actions in competitive sports videos is the key to making predictions accurate and interpretable. We construct a new fine-grained dataset, called FineDiving, developed on diverse diving events with detailed annotations on action procedures.
  arXiv  Detail & Related papers  (2022-04-07T17:59:32Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• Evaluation of Unsupervised Entity and Event Salience Estimation [17.74208462902158]
  Salience Estimation aims to predict term importance in documents. Previous studies typically generate pseudo-ground truth for evaluation. In this work, we propose a light yet practical entity and event salience estimation evaluation protocol.
  arXiv  Detail & Related papers  (2021-04-14T15:23:08Z)
• Aligning Intraobserver Agreement by Transitivity [1.0152838128195467]
  We propose a novel method for measuring within annotator consistency or annotator Intraobserver Agreement (IA) The proposed approach is based on transitivity, a measure that has been thoroughly studied in the context of rational decision-making.
  arXiv  Detail & Related papers  (2020-09-29T09:55:04Z)
• Addressing Class Imbalance in Scene Graph Parsing by Learning to Contrast and Score [65.18522219013786]
  Scene graph parsing aims to detect objects in an image scene and recognize their relations. Recent approaches have achieved high average scores on some popular benchmarks, but fail in detecting rare relations. This paper introduces a novel integrated framework of classification and ranking to resolve the class imbalance problem.
  arXiv  Detail & Related papers  (2020-09-28T13:57:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.