Do We Train on Test Data? The Impact of Near-Duplicates on License Plate Recognition

• URL: http://arxiv.org/abs/2304.04653v2
• Date: Fri, 4 Aug 2023 13:36:29 GMT
• Title: Do We Train on Test Data? The Impact of Near-Duplicates on License Plate Recognition
• Authors: Rayson Laroca, Valter Estevam, Alceu S. Britto Jr., Rodrigo Minetto, David Menotti
• Abstract summary: This work draws attention to the large fraction of near-duplicates in the training and test sets of datasets widely adopted in License Plate Recognition (LPR) research. Our experiments, conducted on the two most popular datasets in the field, show a substantial decrease in recognition rate when six well-known models are trained and tested under fair splits. These findings suggest that such duplicates have significantly biased the evaluation and development of deep learning-based models for LPR.
• Score: 4.6425780769024945
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work draws attention to the large fraction of near-duplicates in the training and test sets of datasets widely adopted in License Plate Recognition (LPR) research. These duplicates refer to images that, although different, show the same license plate. Our experiments, conducted on the two most popular datasets in the field, show a substantial decrease in recognition rate when six well-known models are trained and tested under fair splits, that is, in the absence of duplicates in the training and test sets. Moreover, in one of the datasets, the ranking of models changed considerably when they were trained and tested under duplicate-free splits. These findings suggest that such duplicates have significantly biased the evaluation and development of deep learning-based models for LPR. The list of near-duplicates we have found and proposals for fair splits are publicly available for further research at https://raysonlaroca.github.io/supp/lpr-train-on-test/

Related papers

• Language model developers should report train-test overlap [52.523638165129505]
  We document the practices of 30 model developers, finding that just 9 developers report train-test overlap. We hope our work increases transparency into train-test overlap to increase the community-wide trust in model evaluations.
  arXiv  Detail & Related papers  (2024-10-10T21:44:56Z)
• Localization Is All You Evaluate: Data Leakage in Online Mapping Datasets and How to Fix It [2.1665407462280446]
  State-of-the-art methods are trained predominantly using nuScenes and Argoverse 2 datasets. Over $80$% of nuScenes and $40$% of Argoverse 2 validation and test samples are less than $5$ m from a training sample. We propose geographically disjoint data splits to reveal the true performance in unseen environments.
  arXiv  Detail & Related papers  (2023-12-11T14:43:23Z)
• Leveraging Model Fusion for Improved License Plate Recognition [3.049664973023939]
  fusing multiple models reduces subpar performance on a particular dataset/scenario. Experiments encompass a wide range of datasets, revealing substantial benefits of fusion approaches in both intra- and cross-dataset setups. For applications where the recognition task can tolerate some additional time, though not excessively, an effective strategy is to combine 4-6 models.
  arXiv  Detail & Related papers  (2023-09-08T13:55:16Z)
• A First Look at Dataset Bias in License Plate Recognition [1.8496815029347666]
  dataset bias has been recognized as a severe problem in the computer vision community. This work investigates the dataset bias problem in the License Plate Recognition context.
  arXiv  Detail & Related papers  (2022-08-23T00:20:33Z)
• An advanced combination of semi-supervised Normalizing Flow & Yolo (YoloNF) to detect and recognize vehicle license plates [1.5208105446192792]
  This paper presents a robust and efficient ALPR system based on the state-of-the-art YOLO object detector and Normalizing flows. The model uses two new strategies. Firstly, a two-stage network using YOLO and a normalization flow-based model for normalization to detect Licenses Plates (LP) and recognize the LP with numbers and Arabic characters.
  arXiv  Detail & Related papers  (2022-07-21T22:22:57Z)
• CHALLENGER: Training with Attribution Maps [63.736435657236505]
  We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance. In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.
  arXiv  Detail & Related papers  (2022-05-30T13:34:46Z)
• Multi-Domain Joint Training for Person Re-Identification [51.73921349603597]
  Deep learning-based person Re-IDentification (ReID) often requires a large amount of training data to achieve good performance. It appears that collecting more training data from diverse environments tends to improve the ReID performance. We propose an approach called Domain-Camera-Sample Dynamic network (DCSD) whose parameters can be adaptive to various factors.
  arXiv  Detail & Related papers  (2022-01-06T09:20:59Z)
• On the Cross-dataset Generalization in License Plate Recognition [1.8514314381314887]
  We propose a traditional-split versus leave-one-dataset-out experimental setup to empirically assess the cross-dataset generalization of 12 OCR models. Results shed light on the limitations of the traditional-split protocol for evaluating approaches in the ALPR context.
  arXiv  Detail & Related papers  (2022-01-02T00:56:09Z)
• Deduplicating Training Data Makes Language Models Better [50.22588162039083]
  Existing language modeling datasets contain many near-duplicate examples and long repetitives. Over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data. We develop two tools that allow us to deduplicate training datasets.
  arXiv  Detail & Related papers  (2021-07-14T06:06:52Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• Two-Sample Testing on Ranked Preference Data and the Role of Modeling Assumptions [57.77347280992548]
  In this paper, we design two-sample tests for pairwise comparison data and ranking data. Our test requires essentially no assumptions on the distributions. By applying our two-sample test on real-world pairwise comparison data, we conclude that ratings and rankings provided by people are indeed distributed differently.
  arXiv  Detail & Related papers  (2020-06-21T20:51:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.