Identity Overlap Between Face Recognition Train/Test Data: Causing Optimistic Bias in Accuracy Measurement

• URL: http://arxiv.org/abs/2405.09403v1
• Date: Wed, 15 May 2024 14:59:26 GMT
• Title: Identity Overlap Between Face Recognition Train/Test Data: Causing Optimistic Bias in Accuracy Measurement
• Abstract summary: overlap between training and testing sets causes an optimistic accuracy estimate. Accuracy is commonly estimated as average 10-fold classification accuracy on image pairs from test sets. Our experiments reveal a surprising degree of identity and image overlap between the LFW family of test sets and the MS1MV2 training set.
• Score: 5.3490692492497836
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A fundamental tenet of pattern recognition is that overlap between training and testing sets causes an optimistic accuracy estimate. Deep CNNs for face recognition are trained for N-way classification of the identities in the training set. Accuracy is commonly estimated as average 10-fold classification accuracy on image pairs from test sets such as LFW, CALFW, CPLFW, CFP-FP and AgeDB-30. Because train and test sets have been independently assembled, images and identities in any given test set may also be present in any given training set. In particular, our experiments reveal a surprising degree of identity and image overlap between the LFW family of test sets and the MS1MV2 training set. Our experiments also reveal identity label noise in MS1MV2. We compare accuracy achieved with same-size MS1MV2 subsets that are identity-disjoint and not identity-disjoint with LFW, to reveal the size of the optimistic bias. Using more challenging test sets from the LFW family, we find that the size of the optimistic bias is larger for more challenging test sets. Our results highlight the lack of and the need for identity-disjoint train and test methodology in face recognition research.

Related papers

• Uncertainty-Aware Label Refinement on Hypergraphs for Personalized Federated Facial Expression Recognition [58.98052764581606]
  We develop a novel uncertainty-Aware label refineMent on hYpergraphs (AMY) method. For local training, each local model consists of a backbone, an uncertainty estimation (UE) block, and an expression classification (EC) block. A personalized uncertainty estimator is then introduced to estimate reliable uncertainty weights of samples in the local client.
  arXiv  Detail & Related papers  (2025-01-03T13:59:21Z)
• What is a Goldilocks Face Verification Test Set? [5.518961562474026]
  Face Recognition models are commonly trained with web-scraped datasets containing millions of images. With train and test sets both assembled from web-scraped images, it is critical to ensure disjoint sets of identities between train and test sets. We show that current train and test sets are generally not identity- or even image-disjoint, and that this results in an optimistic bias in the estimated accuracy.
  arXiv  Detail & Related papers  (2024-05-24T22:27:10Z)
• Modeling the Uncertainty with Maximum Discrepant Students for Semi-supervised 2D Pose Estimation [57.17120203327993]
  We propose a framework to estimate the quality of pseudo-labels in semi-supervised pose estimation tasks. Our method improves the performance of semi-supervised pose estimation on three datasets.
  arXiv  Detail & Related papers  (2023-11-03T08:11:06Z)
• Open-set Face Recognition using Ensembles trained on Clustered Data [2.132096006921048]
  This work details a scalable open-set face identification approach to galleries composed of hundreds and thousands of subjects. It is composed of clustering and an ensemble of binary learning algorithms that estimates when query face samples belong to the face gallery and then retrieves their correct identity.
  arXiv  Detail & Related papers  (2023-08-14T20:34:54Z)
• Robustness Stress Testing in Medical Image Classification [26.094688963784254]
  We employ stress testing to assess model robustness and subgroup performance disparities in disease detection models. We apply stress tests to measure the robustness of disease detection models for chest X-ray and skin lesion images. Our experiments indicate that some models may yield more robust and equitable performance than others.
  arXiv  Detail & Related papers  (2023-08-14T02:02:56Z)
• Zero-shot Model Diagnosis [80.36063332820568]
  A common approach to evaluate deep learning models is to build a labeled test set with attributes of interest and assess how well it performs. This paper argues the case that Zero-shot Model Diagnosis (ZOOM) is possible without the need for a test set nor labeling.
  arXiv  Detail & Related papers  (2023-03-27T17:59:33Z)
• Estimating Confidence of Predictions of Individual Classifiers and Their Ensembles for the Genre Classification Task [0.0]
  Genre identification is a subclass of non-topical text classification. Nerve models based on pre-trained transformers, such as BERT or XLM-RoBERTa, demonstrate SOTA results in many NLP tasks.
  arXiv  Detail & Related papers  (2022-06-15T09:59:05Z)
• mc-BEiT: Multi-choice Discretization for Image BERT Pre-training [52.04866462439979]
  Image BERT pre-training with masked image modeling (MIM) is a popular practice to cope with self-supervised representation learning. We introduce an improved BERT-style image pre-training method, namely mc-BEiT, which performs MIM proxy tasks towards eased and refined multi-choice training objectives.
  arXiv  Detail & Related papers  (2022-03-29T09:08:18Z)
• Towards Semi-Supervised Deep Facial Expression Recognition with An Adaptive Confidence Margin [92.76372026435858]
  We learn an Adaptive Confidence Margin (Ada-CM) to fully leverage all unlabeled data for semi-supervised deep facial expression recognition. All unlabeled samples are partitioned into two subsets by comparing their confidence scores with the adaptively learned confidence margin. Our method achieves state-of-the-art performance, especially surpassing fully-supervised baselines in a semi-supervised manner.
  arXiv  Detail & Related papers  (2022-03-23T11:43:29Z)
• Face Identification Proficiency Test Designed Using Item Response Theory [5.284353550094401]
  Triad Identity Matching (TIM) test is based on stimulus difficulty measures based on Item Response Theory (IRT) In Experiment 1, university students showed wide-ranging accuracy on the TIM test. IRT-based item difficulty measures were used to partition the TIM test into three equally "easy" and three equally "difficult" subsets.
  arXiv  Detail & Related papers  (2021-06-22T22:37:32Z)
• Towards Universal Representation Learning for Deep Face Recognition [106.21744671876704]
  We propose a universal representation learning framework that can deal with larger variation unseen in the given training data without leveraging target domain knowledge. Experiments show that our method achieves top performance on general face recognition datasets such as LFW and MegaFace.
  arXiv  Detail & Related papers  (2020-02-26T23:29:57Z)
• Suppressing Uncertainties for Large-Scale Facial Expression Recognition [81.51495681011404]
  This paper proposes a simple yet efficient Self-Cure Network (SCN) which suppresses the uncertainties efficiently and prevents deep networks from over-fitting uncertain facial images. Results on public benchmarks demonstrate that our SCN outperforms current state-of-the-art methods with textbf88.14% on RAF-DB, textbf60.23% on AffectNet, and textbf89.35% on FERPlus.
  arXiv  Detail & Related papers  (2020-02-24T17:24:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.