Data Leakage and Evaluation Issues in Micro-Expression Analysis

• URL: http://arxiv.org/abs/2211.11425v1
• Date: Mon, 21 Nov 2022 13:12:07 GMT
• Title: Data Leakage and Evaluation Issues in Micro-Expression Analysis
• Authors: Tuomas Varanka, Yante Li, Wei Peng and Guoying Zhao
• Abstract summary: We show that data leakage and fragmented evaluation protocols are issues among the micro-expression literature. We propose a new standardized evaluation protocol using facial action units with over 2000 micro-expression samples, and provide an open source library that implements the evaluation protocols in a standardized manner.
• Score: 45.215233522470115
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Micro-expressions have drawn increasing interest lately due to various potential applications. The task is, however, difficult as it incorporates many challenges from the fields of computer vision, machine learning and emotional sciences. Due to the spontaneous and subtle characteristics of micro-expressions, the available training and testing data are limited, which make evaluation complex. We show that data leakage and fragmented evaluation protocols are issues among the micro-expression literature. We find that fixing data leaks can drastically reduce model performance, in some cases even making the models perform similarly to a random classifier. To this end, we go through common pitfalls, propose a new standardized evaluation protocol using facial action units with over 2000 micro-expression samples, and provide an open source library that implements the evaluation protocols in a standardized manner. Code will be available in \url{https://github.com/tvaranka/meb}.

Related papers

• Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks [65.21536453075275]
  We focus on the summarization task and investigate the membership inference (MI) attack. We exploit text similarity and the model's resistance to document modifications as potential MI signals. We discuss several safeguards for training summarization models to protect against MI attacks and discuss the inherent trade-off between privacy and utility.
  arXiv  Detail & Related papers  (2023-10-20T05:44:39Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• Rethinking Dense Retrieval's Few-Shot Ability [24.86681340512899]
  Few-shot dense retrieval aims to generalize to novel search scenarios by learning a few samples. Current methods often resort to random sampling from supervised datasets to create "few-data" setups. We propose a customized FewDR dataset and a unified evaluation benchmark.
  arXiv  Detail & Related papers  (2023-04-12T13:20:16Z)
• CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learning [14.637303913878435]
  We present a coherence-based contrastive learning model named CoCo to detect the possible MGT under low-resource scenario. To exploit the linguistic feature, we encode coherence information in form of graph into text representation. Experiment results on two public datasets and two self-constructed datasets prove our approach outperforms the state-of-art methods significantly.
  arXiv  Detail & Related papers  (2022-12-20T15:26:19Z)
• On the Blind Spots of Model-Based Evaluation Metrics for Text Generation [79.01422521024834]
  We explore a useful but often neglected methodology for robustness analysis of text generation evaluation metrics. We design and synthesize a wide range of potential errors and check whether they result in a commensurate drop in the metric scores. Our experiments reveal interesting insensitivities, biases, or even loopholes in existing metrics.
  arXiv  Detail & Related papers  (2022-12-20T06:24:25Z)
• Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future [63.99570204416711]
  We reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets. We define a uniform evaluation setup including a new formalization of the annotation error detection task. We release our datasets and implementations in an easy-to-use and open source software package.
  arXiv  Detail & Related papers  (2022-06-05T22:31:45Z)
• Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms [52.58031087639394]
  micro-expressions are involuntary and transient facial expressions. They can provide important information in a broad range of applications such as lie detection, criminal detection, etc. Since micro-expressions are transient and of low intensity, their detection and recognition is difficult and relies heavily on expert experiences.
  arXiv  Detail & Related papers  (2022-01-30T05:14:13Z)
• Action Units That Constitute Trainable Micro-expressions (and A Large-scale Synthetic Dataset) [20.866448615388876]
  We aim to develop a protocol to automatically synthesize micro-expression training data on a large scale. Specifically, we discover three types of Action Units (AUs) that can well constitute trainable micro-expressions. With these AUs, our protocol employs large numbers of face images with various identities and an existing face generation method for micro-expression synthesis. Micro-expression recognition models are trained on the generated micro-expression datasets and evaluated on real-world test sets.
  arXiv  Detail & Related papers  (2021-12-03T06:09:06Z)
• Is it feasible to detect FLOSS version release events from textual messages? A case study on Stack Overflow [0.0]
  The study investigates the feasibility of micro-event detection on textual data using a sample of messages from the Stack Overflow Q&A platform. We build pipelines for detection of micro-events using three different estimators whose parameters are optimized using a grid search approach. In our experiments we investigate whether there is a characteristic change in the topics distribution or sentiment features before or after micro-events take place.
  arXiv  Detail & Related papers  (2020-03-30T16:55:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.