What is a Goldilocks Face Verification Test Set?

• URL: http://arxiv.org/abs/2405.15965v1
• Date: Fri, 24 May 2024 22:27:10 GMT
• Title: What is a Goldilocks Face Verification Test Set?
• Abstract summary: Face Recognition models are commonly trained with web-scraped datasets containing millions of images. With train and test sets both assembled from web-scraped images, it is critical to ensure disjoint sets of identities between train and test sets. We show that current train and test sets are generally not identity- or even image-disjoint, and that this results in an optimistic bias in the estimated accuracy.
• Score: 5.518961562474026
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Face Recognition models are commonly trained with web-scraped datasets containing millions of images and evaluated on test sets emphasizing pose, age and mixed attributes. With train and test sets both assembled from web-scraped images, it is critical to ensure disjoint sets of identities between train and test sets. However, existing train and test sets have not considered this. Moreover, as accuracy levels become saturated, such as LFW $>99.8\%$, more challenging test sets are needed. We show that current train and test sets are generally not identity- or even image-disjoint, and that this results in an optimistic bias in the estimated accuracy. In addition, we show that identity-disjoint folds are important in the 10-fold cross-validation estimate of test accuracy. To better support continued advances in face recognition, we introduce two "Goldilocks" test sets, Hadrian and Eclipse. The former emphasizes challenging facial hairstyles and latter emphasizes challenging over- and under-exposure conditions. Images in both datasets are from a large, controlled-acquisition (not web-scraped) dataset, so they are identity- and image-disjoint with all popular training sets. Accuracy for these new test sets generally falls below that observed on LFW, CPLFW, CALFW, CFP-FP and AgeDB-30, showing that these datasets represent important dimensions for improvement of face recognition. The datasets are available at: \url{https://github.com/HaiyuWu/SOTA-Face-Recognition-Train-and-Test}

Related papers

• Pluralistic Salient Object Detection [108.74650817891984]
  We introduce pluralistic salient object detection (PSOD), a novel task aimed at generating multiple plausible salient segmentation results for a given input image. We present two new SOD datasets "DUTS-MM" and "DUS-MQ", along with newly designed evaluation metrics.
  arXiv  Detail & Related papers  (2024-09-04T01:38:37Z)
• Identity Overlap Between Face Recognition Train/Test Data: Causing Optimistic Bias in Accuracy Measurement [5.3490692492497836]
  overlap between training and testing sets causes an optimistic accuracy estimate. Accuracy is commonly estimated as average 10-fold classification accuracy on image pairs from test sets. Our experiments reveal a surprising degree of identity and image overlap between the LFW family of test sets and the MS1MV2 training set.
  arXiv  Detail & Related papers  (2024-05-15T14:59:26Z)
• Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Method [77.65459419417533]
  We put face forgery in a semantic context and define that computational methods that alter semantic face attributes are sources of face forgery. We construct a large face forgery image dataset, where each image is associated with a set of labels organized in a hierarchical graph. We propose a semantics-oriented face forgery detection method that captures label relations and prioritizes the primary task.
  arXiv  Detail & Related papers  (2024-05-14T10:24:19Z)
• Double Trouble? Impact and Detection of Duplicates in Face Image Datasets [7.092869001331781]
  Face image datasets intended for facial biometrics research were created via web-scraping. This work presents an approach to detect both exactly and nearly identical face image duplicates.
  arXiv  Detail & Related papers  (2024-01-25T11:10:13Z)
• Set-Based Face Recognition Beyond Disentanglement: Burstiness Suppression With Variance Vocabulary [78.203301910422]
  We argue that the two crucial issues in SFR, the face quality and burstiness, are both identity-irrelevant and variance-relevant. We propose a light-weighted set-based disentanglement framework to separate the identity features with the variance features. To suppress face burstiness in the sets, we propose a vocabulary-based burst suppression (VBS) method.
  arXiv  Detail & Related papers  (2023-04-13T04:02:58Z)
• Diverse, Difficult, and Odd Instances (D2O): A New Test Set for Object Classification [47.64219291655723]
  We introduce a new test set, called D2O, which is sufficiently different from existing test sets. Our dataset contains 8,060 images spread across 36 categories, out of which 29 appear in ImageNet. The best Top-1 accuracy on our dataset is around 60% which is much lower than 91% best Top-1 accuracy on ImageNet.
  arXiv  Detail & Related papers  (2023-01-29T19:58:32Z)
• Surveillance Face Anti-spoofing [81.50018853811895]
  Face Anti-spoofing (FAS) is essential to secure face recognition systems from various physical attacks. We propose a Contrastive Quality-Invariance Learning (CQIL) network to alleviate the performance degradation caused by image quality. A large number of experiments verify the quality of the SuHiFiMask dataset and the superiority of the proposed CQIL.
  arXiv  Detail & Related papers  (2023-01-03T07:09:57Z)
• When Handcrafted Features and Deep Features Meet Mismatched Training and Test Sets for Deepfake Detection [13.405191042154367]
  In this paper, we utilize and compare two kinds of handcrafted features(SIFT and HoG) and two kinds of deep features(Xception and CNN+RNN) for the deepfake detection task. We also check the performance of these features when there are mismatches between training sets and test sets.
  arXiv  Detail & Related papers  (2022-09-27T10:15:46Z)
• AdaFace: Quality Adaptive Margin for Face Recognition [56.99208144386127]
  We introduce another aspect of adaptiveness in the loss function, namely the image quality. We propose a new loss function that emphasizes samples of different difficulties based on their image quality. Our method, AdaFace, improves the face recognition performance over the state-of-the-art (SoTA) on four datasets.
  arXiv  Detail & Related papers  (2022-04-03T01:23:41Z)
• To Find Waldo You Need Contextual Cues: Debiasing Who's Waldo [53.370023611101175]
  We present a debiased dataset for the Person-centric Visual Grounding task first proposed by Cui et al. Given an image and a caption, PCVG requires pairing up a person's name mentioned in a caption with a bounding box that points to the person in the image. We find that the original Who's Waldo dataset contains a large number of biased samples that are solvable simply by methods.
  arXiv  Detail & Related papers  (2022-03-30T21:35:53Z)
• Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching [53.27673119360868]
  Referring expression grounding is an important and challenging task in computer vision. We propose a novel bidirectional cross-modal matching (BiCM) framework to address these challenges. Our framework outperforms previous works by 6.55% and 9.94% on two popular grounding datasets.
  arXiv  Detail & Related papers  (2022-01-18T01:13:19Z)
• Face Verification with Challenging Imposters and Diversified Demographics [0.0]
  Face verification aims to distinguish between genuine and imposter pairs of faces, which include the same or different identities, respectively. We introduce the $FaVCI2D$ dataset, which includes metadata related to gender, country and age to facilitate fine-grained analysis of results. Experiments with state-of-the-art deep models that provide nearly 100% performance on existing datasets show a significant performance drop for $FaVCI2D$.
  arXiv  Detail & Related papers  (2021-10-16T21:34:59Z)
• Free Lunch for Co-Saliency Detection: Context Adjustment [14.688461235328306]
  We propose a "cost-free" group-cut-paste (GCP) procedure to leverage images from off-the-shelf saliency detection datasets and synthesize new samples. We collect a novel dataset called Context Adjustment Training. The two variants of our dataset, i.e., CAT and CAT+, consist of 16,750 and 33,500 images, respectively.
  arXiv  Detail & Related papers  (2021-08-04T14:51:37Z)
• Face Identification Proficiency Test Designed Using Item Response Theory [5.284353550094401]
  Triad Identity Matching (TIM) test is based on stimulus difficulty measures based on Item Response Theory (IRT) In Experiment 1, university students showed wide-ranging accuracy on the TIM test. IRT-based item difficulty measures were used to partition the TIM test into three equally "easy" and three equally "difficult" subsets.
  arXiv  Detail & Related papers  (2021-06-22T22:37:32Z)
• DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition [85.94331736287765]
  We formulate HFR as a dual generation problem, and tackle it via a novel Dual Variational Generation (DVG-Face) framework. We integrate abundant identity information of large-scale visible data into the joint distribution. Massive new diverse paired heterogeneous images with the same identity can be generated from noises.
  arXiv  Detail & Related papers  (2020-09-20T09:48:24Z)
• A Method for Curation of Web-Scraped Face Image Datasets [13.893682217746816]
  A variety of issues occur when collecting a dataset in-the-wild. With the number of images being in the millions, a manual cleaning procedure is not feasible. We propose a semi-automated method, where the goal is to have a clean dataset for testing face recognition methods.
  arXiv  Detail & Related papers  (2020-04-07T01:57:32Z)
• Evaluating Models' Local Decision Boundaries via Contrast Sets [119.38387782979474]
  We propose a new annotation paradigm for NLP that helps to close systematic gaps in the test data. We demonstrate the efficacy of contrast sets by creating them for 10 diverse NLP datasets. Although our contrast sets are not explicitly adversarial, model performance is significantly lower on them than on the original test sets.
  arXiv  Detail & Related papers  (2020-04-06T14:47:18Z)
• Suppressing Uncertainties for Large-Scale Facial Expression Recognition [81.51495681011404]
  This paper proposes a simple yet efficient Self-Cure Network (SCN) which suppresses the uncertainties efficiently and prevents deep networks from over-fitting uncertain facial images. Results on public benchmarks demonstrate that our SCN outperforms current state-of-the-art methods with textbf88.14% on RAF-DB, textbf60.23% on AffectNet, and textbf89.35% on FERPlus.
  arXiv  Detail & Related papers  (2020-02-24T17:24:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.