When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification

• URL: http://arxiv.org/abs/2505.16149v1
• Date: Thu, 22 May 2025 02:47:36 GMT
• Title: When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identification
• Authors: Zirui Pang, Haosheng Tan, Yuhan Pu, Zhijie Deng, Zhouan Shen, Keyu Hu, Jiaheng Wei,
• Abstract summary: We present a comprehensive framework named REVEAL to address both noisy labels and missing labels in image classification test sets.<n> REVEAL detects potential noisy labels and omissions, aggregates predictions from various methods, and refines label accuracy through confidence-informed predictions and consensus-based filtering.<n>Our method effectively reveals missing labels from public datasets and provides soft-labeled results with likelihoods.
• Score: 11.49089004019603
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Image classification benchmark datasets such as CIFAR, MNIST, and ImageNet serve as critical tools for model evaluation. However, despite the cleaning efforts, these datasets still suffer from pervasive noisy labels and often contain missing labels due to the co-existing image pattern where multiple classes appear in an image sample. This results in misleading model comparisons and unfair evaluations. Existing label cleaning methods focus primarily on noisy labels, but the issue of missing labels remains largely overlooked. Motivated by these challenges, we present a comprehensive framework named REVEAL, integrating state-of-the-art pre-trained vision-language models (e.g., LLaVA, BLIP, Janus, Qwen) with advanced machine/human label curation methods (e.g., Docta, Cleanlab, MTurk), to systematically address both noisy labels and missing label detection in widely-used image classification test sets. REVEAL detects potential noisy labels and omissions, aggregates predictions from various methods, and refines label accuracy through confidence-informed predictions and consensus-based filtering. Additionally, we provide a thorough analysis of state-of-the-art vision-language models and pre-trained image classifiers, highlighting their strengths and limitations within the context of dataset renovation by revealing 10 observations. Our method effectively reveals missing labels from public datasets and provides soft-labeled results with likelihoods. Through human verifications, REVEAL significantly improves the quality of 6 benchmark test sets, highly aligning to human judgments and enabling more accurate and meaningful comparisons in image classification.

Related papers

• Sanitizing Manufacturing Dataset Labels Using Vision-Language Models [1.0819408603463427]
  This paper introduces Vision-Language Sanitization and Refinement (VLSR), which is a vision-language-based framework for label sanitization and refinement.<n>The method embeds both images and their associated textual labels into a shared semantic space leveraging the CLIP vision-language model.<n> Experimental results demonstrate that the VLSR framework successfully identifies problematic labels and improves label consistency.
  arXiv  Detail & Related papers  (2025-06-30T02:13:09Z)
• Estimating label quality and errors in semantic segmentation data via any model [19.84626033109009]
  We study methods to score label quality, such that the images with the lowest scores are least likely to be correctly labeled. This helps prioritize what data to review in order to ensure a high-quality training/evaluation dataset.
  arXiv  Detail & Related papers  (2023-07-11T07:29:09Z)
• Category-Adaptive Label Discovery and Noise Rejection for Multi-label Image Recognition with Partial Positive Labels [78.88007892742438]
  Training multi-label models with partial positive labels (MLR-PPL) attracts increasing attention. Previous works regard unknown labels as negative and adopt traditional MLR algorithms. We propose to explore semantic correlation among different images to facilitate the MLR-PPL task.
  arXiv  Detail & Related papers  (2022-11-15T02:11:20Z)
• Dual-Perspective Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels [70.36722026729859]
  We propose a dual-perspective semantic-aware representation blending (DSRB) that blends multi-granularity category-specific semantic representation across different images. The proposed DS consistently outperforms current state-of-the-art algorithms on all proportion label settings.
  arXiv  Detail & Related papers  (2022-05-26T00:33:44Z)
• Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels [86.17081952197788]
  We propose to blend category-specific representation across different images to transfer information of known labels to complement unknown labels. Experiments on the MS-COCO, Visual Genome, Pascal VOC 2007 datasets show that the proposed SARB framework obtains superior performance over current leading competitors.
  arXiv  Detail & Related papers  (2022-03-04T07:56:16Z)
• BoMD: Bag of Multi-label Descriptors for Noisy Chest X-ray Classification [25.76256302330625]
  New medical imaging classification problems may need to rely on machine-generated noisy labels extracted from radiology reports. Current noisy-label learning methods designed for multi-class problems cannot be easily adapted. We propose a new method designed for the noisy multi-label CXR learning, which detects and smoothly re-labels samples from the dataset.
  arXiv  Detail & Related papers  (2022-03-03T08:04:59Z)
• Learning to Aggregate and Refine Noisy Labels for Visual Sentiment Analysis [69.48582264712854]
  We propose a robust learning method to perform robust visual sentiment analysis. Our method relies on an external memory to aggregate and filter noisy labels during training. We establish a benchmark for visual sentiment analysis with label noise using publicly available datasets.
  arXiv  Detail & Related papers  (2021-09-15T18:18:28Z)
• Active label cleaning: Improving dataset quality under resource constraints [13.716577886649018]
  Imperfections in data annotation, known as label noise, are detrimental to the training of machine learning models. This work advocates for a data-driven approach to prioritising samples for re-annotation. We rank instances according to estimated label correctness and labelling difficulty of each sample, and introduce a simulation framework to evaluate relabelling efficacy.
  arXiv  Detail & Related papers  (2021-09-01T19:03:57Z)
• Learning from Pixel-Level Label Noise: A New Perspective for Semi-Supervised Semantic Segmentation [12.937770890847819]
  We propose a graph based label noise detection and correction framework to deal with pixel-level noisy labels. In particular, for the generated pixel-level noisy labels from weak supervisions by Class Activation Map (CAM), we train a clean segmentation model with strong supervisions. Finally, we adopt a superpixel-based graph to represent the relations of spatial adjacency and semantic similarity between pixels in one image.
  arXiv  Detail & Related papers  (2021-03-26T03:23:21Z)
• Attention-Aware Noisy Label Learning for Image Classification [97.26664962498887]
  Deep convolutional neural networks (CNNs) learned on large-scale labeled samples have achieved remarkable progress in computer vision. The cheapest way to obtain a large body of labeled visual data is to crawl from websites with user-supplied labels, such as Flickr. This paper proposes the attention-aware noisy label learning approach to improve the discriminative capability of the network trained on datasets with potential label noise.
  arXiv  Detail & Related papers  (2020-09-30T15:45:36Z)
• Learning Image Labels On-the-fly for Training Robust Classification Models [13.669654965671604]
  We show how noisy annotations (e.g., from different algorithm-based labelers) can be utilized together and mutually benefit the learning of classification tasks. A meta-training based label-sampling module is designed to attend the labels that benefit the model learning the most through additional back-propagation processes.
  arXiv  Detail & Related papers  (2020-09-22T05:38:44Z)
• Unsupervised Person Re-identification via Multi-label Classification [55.65870468861157]
  This paper formulates unsupervised person ReID as a multi-label classification task to progressively seek true labels. Our method starts by assigning each person image with a single-class label, then evolves to multi-label classification by leveraging the updated ReID model for label prediction. To boost the ReID model training efficiency in multi-label classification, we propose the memory-based multi-label classification loss (MMCL)
  arXiv  Detail & Related papers  (2020-04-20T12:13:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.