Beyond Error-vs-Discard Characteristic: Toward Stable and Reliable Evaluation for Face Image Quality Assessment

• URL: http://arxiv.org/abs/2607.22752v1
• Date: Thu, 23 Jul 2026 14:38:59 GMT
• Title: Beyond Error-vs-Discard Characteristic: Toward Stable and Reliable Evaluation for Face Image Quality Assessment
• Abstract summary: Face Image Quality Assessment (FIQA) aims to estimate the utility of facial images for reliable recognition.<n>Error-versus-Discard Characteristic (EDC) evaluates performance by progressively discarding low-quality samples and measuring recognition error on the retained subset.<n>In this work, we demonstrate that the widely used EDC protocol has fundamental limitations: Test-Set Divergence and Threshold Drift.
• Score: 1.9395146971956156
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Face Image Quality Assessment (FIQA) aims to estimate the utility of facial images for reliable recognition. The evaluation of FIQA methods is predominantly based on the Error-versus-Discard Characteristic (EDC), which evaluates performance by progressively discarding low-quality samples and measuring recognition error on the retained subset. In this work, we demonstrate that the widely used EDC protocol has fundamental limitations: Test-Set Divergence and Threshold Drift, which together limit the reliability and comparability of FIQA methods. To address this, we propose discard-based EDC variants and a rank-based Rank Consistency Evaluation (RCE) metric that operates on the entire test set without discarding samples, using a fixed decision threshold. Extensive experiments on five datasets, four face recognition models, and 15 state-of-the-art FIQA methods demonstrate both the limitations of EDC and the effectiveness of the proposed approaches in enabling a more reliable and comparable evaluation. Despite evaluated on face images only, the limitations arise from the EDC protocol rather than the biometric modality, suggesting a broader applicability to biometric quality assessment in general.

Related papers

• Q-DeepSight: Incentivizing Thinking with Images for Image Quality Assessment and Refinement [58.15004031934379]
  We propose Q-DeepSight, a think-with-image framework that emulates this human-like process.<n>Q-DeepSight achieves state-of-the-art performance across diverse benchmarks, including natural, restored, and AI-generated content.<n>We demonstrate its practical value with Perceptual-in-Generation (PiG), a training-free framework where Q-DeepSight's diagnoses guide iterative image enhancement.
  arXiv  Detail & Related papers  (2026-04-18T06:10:57Z)
• Surveillance Facial Image Quality Assessment: A Multi-dimensional Dataset and Lightweight Model [59.39390911456143]
  We propose the first comprehensive study on surveillance facial image quality assessment (SFIQA)<n>SFIQA-Bench consists of 5,004 surveillance facial images captured by three widely deployed surveillance cameras in real-world scenarios.<n>A subjective experiment is conducted to collect six dimensional quality ratings, including noise, sharpness, colorfulness, contrast, fidelity and overall quality.
  arXiv  Detail & Related papers  (2026-02-07T06:51:03Z)
• Fine-Grained Human Pose Editing Assessment via Layer-Selective MLLMs [70.31435391393642]
  We introduce a benchmark comprising 1,700 standardized samples from 17 state-of-the-art editing models.<n>We propose a unified framework based on layer-selective multimodal large language models (MLLMs)<n>Our framework achieves superior performance in both authenticity detection and multi-dimensional quality regression.
  arXiv  Detail & Related papers  (2026-01-15T13:22:07Z)
• FROQ: Observing Face Recognition Models for Efficient Quality Assessment [5.154970335677459]
  Face Recognition (FR) plays a crucial role in many critical (high-stakes) applications.<n>Most state-of-the-art FIQA techniques rely on extensive supervised training to achieve accurate quality estimation.<n>In this paper, we introduce FROQ, a semi-supervised, training-free approach to estimate face-image quality.
  arXiv  Detail & Related papers  (2025-09-22T12:29:44Z)
• CDI: Blind Image Restoration Fidelity Evaluation based on Consistency with Degraded Image [6.664998519426364]
  We reassess the Solution Non-Uniqueness and Degradation Indeterminacy issues of Blind Image Restoration (BIR) methods.<n>We propose a specific BIR IQA system, which evaluates fidelity by calculating the Consistency with Degraded Image (CDI)<n>In addition, we propose a Reference Agnostic CDI, enabling BIR fidelity evaluation without reference images.
  arXiv  Detail & Related papers  (2025-01-24T06:05:47Z)
• Position: AI Evaluation Should Learn from How We Test Humans [65.36614996495983]
  We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• DifFIQA: Face Image Quality Assessment Using Denoising Diffusion Probabilistic Models [1.217503190366097]
  Face image quality assessment (FIQA) techniques aim to mitigate these performance degradations. We present a powerful new FIQA approach, named DifFIQA, which relies on denoising diffusion probabilistic models (DDPM) Because the diffusion-based perturbations are computationally expensive, we also distill the knowledge encoded in DifFIQA into a regression-based quality predictor, called DifFIQA(R)
  arXiv  Detail & Related papers  (2023-05-09T21:03:13Z)
• Considerations on the Evaluation of Biometric Quality Assessment Algorithms [7.092869001331781]
  Quality assessment algorithms can be used to estimate the utility of a biometric sample for the purpose of biometric recognition. "Error versus Characteristic" (EDC) plots, and "partial Area Under Curve" (pAUC) values of curves therein, are generally used by researchers to evaluate such quality assessment algorithms.
  arXiv  Detail & Related papers  (2023-03-23T14:26:21Z)
• Fingerprint Image-Quality Estimation and its Application to Multialgorithm Verification [56.128200319868526]
  Signal-quality awareness has been found to increase recognition rates and to support decisions in multisensor environments significantly. Here, we study the orientation tensor of fingerprint images to quantify signal impairments, such as noise, lack of structure, blur, with the help of symmetry descriptors. The quantitative results favor quality awareness under all aspects, boosting recognition rates and fusing differently skilled experts efficiently as well as effectively.
  arXiv  Detail & Related papers  (2022-11-24T12:17:49Z)
• Uncertainty-Driven Action Quality Assessment [11.958132175629368]
  We propose a novel probabilistic model, named Uncertainty-Driven AQA (UD-AQA), to capture the diversity among multiple judge scores.<n>We generate the estimation of uncertainty for each prediction, which is employed to re-weight AQA regression loss.<n>Our proposed method achieves competitive results on three benchmarks including the Olympic events MTL-AQA and FineDiving, and the surgical skill JIGSAWS datasets.
  arXiv  Detail & Related papers  (2022-07-29T07:21:15Z)
• Assessment of Treatment Effect Estimators for Heavy-Tailed Data [70.72363097550483]
  A central obstacle in the objective assessment of treatment effect (TE) estimators in randomized control trials (RCTs) is the lack of ground truth (or validation set) to test their performance. We provide a novel cross-validation-like methodology to address this challenge. We evaluate our methodology across 709 RCTs implemented in the Amazon supply chain.
  arXiv  Detail & Related papers  (2021-12-14T17:53:01Z)
• FaceQvec: Vector Quality Assessment for Face Biometrics based on ISO Compliance [15.913755899679733]
  FaceQvec is a software component for estimating the conformity of facial images with each of the points contemplated in the ISO/IEC 19794-5. This quality standard defines general quality guidelines for face images that would make them acceptable or unacceptable for use in official documents such as passports or ID cards.
  arXiv  Detail & Related papers  (2021-11-03T09:07:41Z)
• Performance Evaluation of Adversarial Attacks: Discrepancies and Solutions [51.8695223602729]
  adversarial attack methods have been developed to challenge the robustness of machine learning models. We propose a Piece-wise Sampling Curving (PSC) toolkit to effectively address the discrepancy. PSC toolkit offers options for balancing the computational cost and evaluation effectiveness.
  arXiv  Detail & Related papers  (2021-04-22T14:36:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.