Evaluating Out-of-Distribution Performance on Document Image Classifiers

• URL: http://arxiv.org/abs/2210.07448v1
• Date: Fri, 14 Oct 2022 01:24:21 GMT
• Title: Evaluating Out-of-Distribution Performance on Document Image Classifiers
• Abstract summary: The RVL-CDIP corpus is the de facto standard benchmark for document classification. All studies that use this corpus do not include evaluation on out-of-distribution documents. Our new benchmark provides researchers with a valuable new resource for analyzing out-of-distribution document performance.
• Score: 2.9962867693321473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The ability of a document classifier to handle inputs that are drawn from a distribution different from the training distribution is crucial for robust deployment and generalizability. The RVL-CDIP corpus is the de facto standard benchmark for document classification, yet to our knowledge all studies that use this corpus do not include evaluation on out-of-distribution documents. In this paper, we curate and release a new out-of-distribution benchmark for evaluating out-of-distribution performance for document classifiers. Our new out-of-distribution benchmark consists of two types of documents: those that are not part of any of the 16 in-domain RVL-CDIP categories (RVL-CDIP-O), and those that are one of the 16 in-domain categories yet are drawn from a distribution different from that of the original RVL-CDIP dataset (RVL-CDIP-N). While prior work on document classification for in-domain RVL-CDIP documents reports high accuracy scores, we find that these models exhibit accuracy drops of between roughly 15-30% on our new out-of-domain RVL-CDIP-N benchmark, and further struggle to distinguish between in-domain RVL-CDIP-N and out-of-domain RVL-CDIP-O inputs. Our new benchmark provides researchers with a valuable new resource for analyzing out-of-distribution performance on document classifiers. Our new out-of-distribution data can be found at https://tinyurl.com/4he6my23.

Related papers

• Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap [3.4560801215716648]
  We produce several variations of RVL-CDIP with label error and test-train overlap fixes, and benchmark document classification performance on these new variations.<n>Our rigorous analysis of RVL-CDIP finds that the corpus contains 12% label error and approximately 35% test-train duplication.<n>We additionally evaluate models on RVL-CDIP-N, an out-of-distribution benchmark, finding that training on error-corrected data substantially improves OOD generalization.
  arXiv  Detail & Related papers  (2026-06-30T10:22:50Z)
• Improving OCR using internal document redundancy [5.123479119457136]
  We propose an unsupervised method by leveraging the redundancy of character shapes within a document to correct imperfect outputs of a given OCR system.<n>We demonstrate improvements in documents with various levels of degradation, including recovered Uruguayan military archives and 17th to mid-20th century European newspapers.
  arXiv  Detail & Related papers  (2025-08-20T09:21:43Z)
• On Evaluation of Document Classification using RVL-CDIP [3.575650613011277]
  The RVL-CDIP benchmark is widely used for measuring performance on the task of document classification. We reveal several undesirable characteristics of the RVL-CDIP benchmark. We advocate for the creation of a new document classification benchmark, and provide recommendations for what characteristics such a resource should include.
  arXiv  Detail & Related papers  (2023-06-21T20:32:22Z)
• Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Models [68.12229916000584]
  We develop an out-of-distribution (OOD) benchmark termed Do-GOOD for the fine-Grained analysis on Document image-related tasks. We then evaluate the robustness and perform a fine-grained analysis of 5 latest VDU pre-trained models and 2 typical OOD generalization algorithms.
  arXiv  Detail & Related papers  (2023-06-05T06:50:42Z)
• Revisiting Long-tailed Image Classification: Survey and Benchmarks with New Evaluation Metrics [88.39382177059747]
  A corpus of metrics is designed for measuring the accuracy, robustness, and bounds of algorithms for learning with long-tailed distribution. Based on our benchmarks, we re-evaluate the performance of existing methods on CIFAR10 and CIFAR100 datasets.
  arXiv  Detail & Related papers  (2023-02-03T02:40:54Z)
• An Upper Bound for the Distribution Overlap Index and Its Applications [18.481370450591317]
  This paper proposes an easy-to-compute upper bound for the overlap index between two probability distributions. The proposed bound shows its value in one-class classification and domain shift analysis. Our work shows significant promise toward broadening the applications of overlap-based metrics.
  arXiv  Detail & Related papers  (2022-12-16T20:02:03Z)
• Explaining Cross-Domain Recognition with Interpretable Deep Classifier [100.63114424262234]
  Interpretable Deep (IDC) learns the nearest source samples of a target sample as evidence upon which the classifier makes the decision. Our IDC leads to a more explainable model with almost no accuracy degradation and effectively calibrates classification for optimum reject options.
  arXiv  Detail & Related papers  (2022-11-15T15:58:56Z)
• Few-Shot Document-Level Relation Extraction [0.0]
  We present document-level relation extraction benchmark (FSDLRE) We argue that document-level corpora provide more realism, particularly regarding none-of-the-above (NOTA) distributions. We adapt the state-of-the-art sentence-level method MNAV to the document-level and develop it further for improved domain adaptation.
  arXiv  Detail & Related papers  (2022-05-04T13:16:19Z)
• Out-of-Category Document Identification Using Target-Category Names as Weak Supervision [64.671654559798]
  Out-of-category detection aims to distinguish documents according to their semantic relevance to the inlier (or target) categories. We present an out-of-category detection framework, which effectively measures how confidently each document belongs to one of the target categories.
  arXiv  Detail & Related papers  (2021-11-24T21:01:25Z)
• Exploring Out-of-Distribution Generalization in Text Classifiers Trained on Tobacco-3482 and RVL-CDIP [0.6871147064097144]
  This paper explores the ability of text classifiers trained on standard document classification datasets to generalize to out-of-distribution documents at inference time. We take the Tobacco-3482 and RVL-CDIP datasets as a starting point and generate new out-of-distribution evaluation datasets. We find that models trained on the smaller Tobacco-3482 dataset perform poorly on our new out-of-distribution data, while text classification models trained on the larger RVL-CDIP exhibit smaller performance drops.
  arXiv  Detail & Related papers  (2021-08-05T15:34:56Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)
• Towards Fair Cross-Domain Adaptation via Generative Learning [50.76694500782927]
  Domain Adaptation (DA) targets at adapting a model trained over the well-labeled source domain to the unlabeled target domain lying in different distributions. We develop a novel Generative Few-shot Cross-domain Adaptation (GFCA) algorithm for fair cross-domain classification.
  arXiv  Detail & Related papers  (2020-03-04T23:25:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.