On Evaluation of Document Classification using RVL-CDIP

• URL: http://arxiv.org/abs/2306.12550v1
• Date: Wed, 21 Jun 2023 20:32:22 GMT
• Title: On Evaluation of Document Classification using RVL-CDIP
• Abstract summary: The RVL-CDIP benchmark is widely used for measuring performance on the task of document classification. We reveal several undesirable characteristics of the RVL-CDIP benchmark. We advocate for the creation of a new document classification benchmark, and provide recommendations for what characteristics such a resource should include.
• Score: 3.575650613011277
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The RVL-CDIP benchmark is widely used for measuring performance on the task of document classification. Despite its widespread use, we reveal several undesirable characteristics of the RVL-CDIP benchmark. These include (1) substantial amounts of label noise, which we estimate to be 8.1% (ranging between 1.6% to 16.9% per document category); (2) presence of many ambiguous or multi-label documents; (3) a large overlap between test and train splits, which can inflate model performance metrics; and (4) presence of sensitive personally-identifiable information like US Social Security numbers (SSNs). We argue that there is a risk in using RVL-CDIP for benchmarking document classifiers, as its limited scope, presence of errors (state-of-the-art models now achieve accuracy error rates that are within our estimated label error rate), and lack of diversity make it less than ideal for benchmarking. We further advocate for the creation of a new document classification benchmark, and provide recommendations for what characteristics such a resource should include.

Related papers

• Benchmarking Classical and Transformer-Based Models for Document Sensitivity Classification [1.096626056612224]
  This paper introduces Strategic 16K, a leakage-controlled corpus of 16,000 diplomatic cables sourced from the WikiLeaks Public Library of US Diplomacy (PlusD)<n>We document an extended leakage removal protocol that identifies and eliminates three categories of residual classification markers embedded within document bodies.<n>Among classical models, TF-IDF with Logistic Regression achieves the strongest performance at significantly lower computational cost.
  arXiv  Detail & Related papers  (2026-08-05T05:00:41Z)
• Neuro-Symbolic AI for LEED compliance: Document-Centric Benchmarking, Deterministic Numeric Checking, and When Multimodal Hurts [0.0]
  This paper investigates whether small, locally deployed language models can perform meaningful screening of LEED documentation.<n>A neuro-symbolic pipeline is introduced that aligns project PDFs to LEED credit sections, retrieves evidence with credit-aware keyword signatures, and applies a LEED-specific numeric checker to quantitative thresholds.
  arXiv  Detail & Related papers  (2026-07-17T05:45:02Z)
• Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap [3.4560801215716648]
  We produce several variations of RVL-CDIP with label error and test-train overlap fixes, and benchmark document classification performance on these new variations.<n>Our rigorous analysis of RVL-CDIP finds that the corpus contains 12% label error and approximately 35% test-train duplication.<n>We additionally evaluate models on RVL-CDIP-N, an out-of-distribution benchmark, finding that training on error-corrected data substantially improves OOD generalization.
  arXiv  Detail & Related papers  (2026-06-30T10:22:50Z)
• Learning More from Less: Unlocking Internal Representations for Benchmark Compression [37.69575776639016]
  We introduce REPCORE, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets.<n>Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy.
  arXiv  Detail & Related papers  (2026-01-31T13:11:39Z)
• Beyond the Numbers: Transparency in Relation Extraction Benchmark Creation and Leaderboards [5.632231145349045]
  This paper investigates the transparency in the creation of benchmarks and the use of leaderboards for measuring progress in NLP. Existing relation extraction benchmarks often suffer from insufficient documentation and lack crucial details. While our discussion centers on the transparency of RE benchmarks and leaderboards, the observations we discuss are broadly applicable to other NLP tasks as well.
  arXiv  Detail & Related papers  (2024-11-07T22:36:19Z)
• Multiclass ROC [6.941573057921458]
  We provide an evaluation metric summarizing the pair-wise multi-class True Positive Rate (TPR) and False Positive Rate (FPR) An integration over those factorized vector provides a binary AUC-equivalent summary on the performance. To support our findings, we conducted extensive simulation studies and compared our method to the pair-wise averaged AUC statistics on benchmark datasets.
  arXiv  Detail & Related papers  (2024-04-19T19:25:10Z)
• Adaptive Hierarchical Certification for Segmentation using Randomized Smoothing [87.48628403354351]
  certification for machine learning is proving that no adversarial sample can evade a model within a range under certain conditions. Common certification methods for segmentation use a flat set of fine-grained classes, leading to high abstain rates due to model uncertainty. We propose a novel, more practical setting, which certifies pixels within a multi-level hierarchy, and adaptively relaxes the certification to a coarser level for unstable components.
  arXiv  Detail & Related papers  (2024-02-13T11:59:43Z)
• Towards Multiple References Era -- Addressing Data Leakage and Limited Reference Diversity in NLG Evaluation [55.92852268168816]
  N-gram matching-based evaluation metrics, such as BLEU and chrF, are widely utilized across a range of natural language generation (NLG) tasks. Recent studies have revealed a weak correlation between these matching-based metrics and human evaluations. We propose to utilize textitmultiple references to enhance the consistency between these metrics and human evaluations.
  arXiv  Detail & Related papers  (2023-08-06T14:49:26Z)
• Characterizing the Optimal 0-1 Loss for Multi-class Classification with a Test-time Attacker [57.49330031751386]
  We find achievable information-theoretic lower bounds on loss in the presence of a test-time attacker for multi-class classifiers on any discrete dataset. We provide a general framework for finding the optimal 0-1 loss that revolves around the construction of a conflict hypergraph from the data and adversarial constraints.
  arXiv  Detail & Related papers  (2023-02-21T15:17:13Z)
• An Upper Bound for the Distribution Overlap Index and Its Applications [22.92968284023414]
  This paper proposes an easy-to-compute upper bound for the overlap index between two probability distributions. The proposed bound shows its value in one-class classification and domain shift analysis. Our work shows significant promise toward broadening the applications of overlap-based metrics.
  arXiv  Detail & Related papers  (2022-12-16T20:02:03Z)
• Evaluating and Improving Factuality in Multimodal Abstractive Summarization [91.46015013816083]
  We propose CLIPBERTScore to leverage the robustness and strong factuality detection performance between image-summary and document-summary. We show that this simple combination of two metrics in the zero-shot achieves higher correlations than existing factuality metrics for document summarization. Our analysis demonstrates the robustness and high correlation of CLIPBERTScore and its components on four factuality metric-evaluation benchmarks.
  arXiv  Detail & Related papers  (2022-11-04T16:50:40Z)
• Evaluating Out-of-Distribution Performance on Document Image Classifiers [2.9962867693321473]
  The RVL-CDIP corpus is the de facto standard benchmark for document classification. All studies that use this corpus do not include evaluation on out-of-distribution documents. Our new benchmark provides researchers with a valuable new resource for analyzing out-of-distribution document performance.
  arXiv  Detail & Related papers  (2022-10-14T01:24:21Z)
• Out-of-Category Document Identification Using Target-Category Names as Weak Supervision [64.671654559798]
  Out-of-category detection aims to distinguish documents according to their semantic relevance to the inlier (or target) categories. We present an out-of-category detection framework, which effectively measures how confidently each document belongs to one of the target categories.
  arXiv  Detail & Related papers  (2021-11-24T21:01:25Z)
• Low-Shot Validation: Active Importance Sampling for Estimating Classifier Performance on Rare Categories [47.050853657721596]
  For machine learning models trained with limited labeled training data, validation stands to become the main bottleneck to reducing overall annotation costs. We propose a statistical validation algorithm that accurately estimates the F-score of binary classifiers for rare categories. In particular, we can estimate model F1 scores with a variance of 0.005 using as few as 100 labels.
  arXiv  Detail & Related papers  (2021-09-13T06:01:16Z)
• Active Bayesian Assessment for Black-Box Classifiers [20.668691047355072]
  We introduce an active Bayesian approach for assessment of classifier performance to satisfy the desiderata of both reliability and label-efficiency. We first develop inference strategies to quantify uncertainty for common assessment metrics such as accuracy, misclassification cost, and calibration error. We then propose a general framework for active Bayesian assessment using inferred uncertainty to guide efficient selection of instances for labeling.
  arXiv  Detail & Related papers  (2020-02-16T08:08:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.