Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap

• URL: http://arxiv.org/abs/2606.31446v1
• Date: Tue, 30 Jun 2026 10:22:50 GMT
• Title: Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap
• Abstract summary: We produce several variations of RVL-CDIP with label error and test-train overlap fixes, and benchmark document classification performance on these new variations.<n>Our rigorous analysis of RVL-CDIP finds that the corpus contains 12% label error and approximately 35% test-train duplication.<n>We additionally evaluate models on RVL-CDIP-N, an out-of-distribution benchmark, finding that training on error-corrected data substantially improves OOD generalization.
• Score: 3.4560801215716648
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: RVL-CDIP is a popular dataset for benchmarking document classifiers. However, the dataset contains ample amounts of label errors as well as non-trivial amounts of test-train overlap, both of which may impact model performance metrics. In this paper, we address these two problems by (1) finding and fixing label errors, and (2) detecting and addressing test-train overlap. We produce several variations of RVL-CDIP with label error and test-train overlap fixes, and benchmark document classification performance on these new RVL-CDIP variations. Our rigorous analysis of RVL-CDIP finds that the corpus contains 12\% label error and approximately 35% test-train duplication. Remediation sees improvements in classification accuracy when errors are removed, but sees decreases in accuracy when duplicates are removed. We additionally evaluate models on RVL-CDIP-N, an out-of-distribution benchmark, finding that training on error-corrected data substantially improves OOD generalization, with supervised models gaining an average of 8.1 percentage points in accuracy and improvements as large as 14 percentage points.

Related papers

• Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning [53.9713678229744]
  Multi-instance partial-label learning (MIPL) is a weakly supervised framework that addresses the challenges of inexact supervision in both instance and label spaces.<n>Existing MIPL approaches often suffer from poor calibration, undermining reliability.<n>We propose a plug-and-play calibratable disambiguation loss (CDL) that simultaneously improves classification accuracy and calibration performance.
  arXiv  Detail & Related papers  (2025-12-19T16:58:31Z)
• From Label Error Detection to Correction: A Modular Framework and Benchmark for Object Detection Datasets [4.864032555684836]
  We introduce a semi-automated framework for label-error correction called REC$checkmark$D (Rechecked)<n>We show that current label error detection methods, when combined with our correction framework, can recover hundreds of errors in the time it would take a human to annotate bounding boxes from scratch.<n>This validated set will be released as a new real-world benchmark for label error detection and correction.
  arXiv  Detail & Related papers  (2025-08-06T10:03:05Z)
• CorBenchX: Large-Scale Chest X-Ray Error Dataset and Vision-Language Model Benchmark for Report Error Correction [11.731590131260424]
  CorBenchX is a suite for automated error detection and correction in chest X-ray reports.<n>We first synthesize a large-scale dataset of 26,326 chest X-ray error reports.<n>We benchmark both open- and closed-source vision-language models.
  arXiv  Detail & Related papers  (2025-05-17T15:39:39Z)
• Tgea: An error-annotated dataset and benchmark tasks for text generation from pretrained language models [57.758735361535486]
  TGEA is an error-annotated dataset for text generation from pretrained language models (PLMs)<n>We create an error taxonomy to cover 24 types of errors occurring in PLM-generated sentences.<n>This is the first dataset with comprehensive annotations for PLM-generated texts.
  arXiv  Detail & Related papers  (2025-03-06T09:14:02Z)
• Teaching Your Models to Understand Code via Focal Preference Alignment [70.71693365502212]
  In existing approaches, a set of n candidate solutions is evaluated based on test case success rates.<n>Because this approach aligns entire failing code blocks rather than pinpointing specific errors, it lacks the granularity necessary to capture meaningful error-correction relationships.<n>We propose Target-DPO, a new preference alignment framework that mimics human iterative debug to refine Code LLMs.
  arXiv  Detail & Related papers  (2025-03-04T16:56:34Z)
• On Evaluation of Document Classification using RVL-CDIP [3.575650613011277]
  The RVL-CDIP benchmark is widely used for measuring performance on the task of document classification. We reveal several undesirable characteristics of the RVL-CDIP benchmark. We advocate for the creation of a new document classification benchmark, and provide recommendations for what characteristics such a resource should include.
  arXiv  Detail & Related papers  (2023-06-21T20:32:22Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• Revisiting Long-tailed Image Classification: Survey and Benchmarks with New Evaluation Metrics [88.39382177059747]
  A corpus of metrics is designed for measuring the accuracy, robustness, and bounds of algorithms for learning with long-tailed distribution. Based on our benchmarks, we re-evaluate the performance of existing methods on CIFAR10 and CIFAR100 datasets.
  arXiv  Detail & Related papers  (2023-02-03T02:40:54Z)
• Evaluating Out-of-Distribution Performance on Document Image Classifiers [2.9962867693321473]
  The RVL-CDIP corpus is the de facto standard benchmark for document classification. All studies that use this corpus do not include evaluation on out-of-distribution documents. Our new benchmark provides researchers with a valuable new resource for analyzing out-of-distribution document performance.
  arXiv  Detail & Related papers  (2022-10-14T01:24:21Z)
• CTRL: Clustering Training Losses for Label Error Detection [4.49681473359251]
  In supervised machine learning, use of correct labels is extremely important to ensure high accuracy. We propose a novel framework, calledClustering TRaining Losses for label error detection. It detects label errors in two steps based on the observation that models learn clean and noisy labels in different ways.
  arXiv  Detail & Related papers  (2022-08-17T18:09:19Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.