Avoid Wasted Annotation Costs in Open-set Active Learning with Pre-trained Vision-Language Model

• URL: http://arxiv.org/abs/2408.04917v1
• Date: Fri, 9 Aug 2024 07:54:57 GMT
• Title: Avoid Wasted Annotation Costs in Open-set Active Learning with Pre-trained Vision-Language Model
• Authors: Jaehyuk Heo, Pilsung Kang,
• Abstract summary: Active learning (AL) aims to enhance model performance by selectively collecting highly informative data. In practical scenarios, unlabeled data may contain out-of-distribution (OOD) samples, leading to wasted annotation costs. We propose a novel selection strategy, CLIPN for AL (CLIPNAL), which minimizes cost losses without requiring OOD samples.
• Score: 3.647905567437244
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Active learning (AL) aims to enhance model performance by selectively collecting highly informative data, thereby minimizing annotation costs. However, in practical scenarios, unlabeled data may contain out-of-distribution (OOD) samples, leading to wasted annotation costs if data is incorrectly selected. Recent research has explored methods to apply AL to open-set data, but these methods often require or incur unavoidable cost losses to minimize them. To address these challenges, we propose a novel selection strategy, CLIPN for AL (CLIPNAL), which minimizes cost losses without requiring OOD samples. CLIPNAL sequentially evaluates the purity and informativeness of data. First, it utilizes a pre-trained vision-language model to detect and exclude OOD data by leveraging linguistic and visual information of in-distribution (ID) data without additional training. Second, it selects highly informative data from the remaining ID data, and then the selected samples are annotated by human experts. Experimental results on datasets with various open-set conditions demonstrate that CLIPNAL achieves the lowest cost loss and highest performance across all scenarios. Code is available at https://github.com/DSBA-Lab/OpenAL.

Related papers

• Compute-Constrained Data Selection [77.06528009072967]
  We formalize the problem of data selection with a cost-aware utility function, and model the problem as trading off initial-selection cost for training gain. We run a comprehensive sweep of experiments across multiple tasks, varying compute budget by scaling finetuning tokens, model sizes, and data selection compute.
  arXiv  Detail & Related papers  (2024-10-21T17:11:21Z)
• A CLIP-Powered Framework for Robust and Generalizable Data Selection [51.46695086779598]
  Real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance. Data selection has shown promise in identifying the most representative samples from the entire dataset. We propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection.
  arXiv  Detail & Related papers  (2024-10-15T03:00:58Z)
• DRUPI: Dataset Reduction Using Privileged Information [20.59889438709671]
  dataset reduction (DR) seeks to select or distill samples from large datasets into smaller subsets while preserving performance on target tasks. We introduce dataset Reduction Using Privileged Information (DRUPI), which enriches DR by synthesizing privileged information alongside the reduced dataset. Our findings reveal that effective feature labels must balance between being overly discriminative and excessively diverse, with a moderate level proving optimal for improving the reduced dataset's efficacy.
  arXiv  Detail & Related papers  (2024-10-02T14:49:05Z)
• XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learners [71.8257151788923]
  We propose a novel Explainable Active Learning framework (XAL) for low-resource text classification. XAL encourages classifiers to justify their inferences and delve into unlabeled data for which they cannot provide reasonable explanations. Experiments on six datasets show that XAL achieves consistent improvement over 9 strong baselines.
  arXiv  Detail & Related papers  (2023-10-09T08:07:04Z)
• Adaptive Negative Evidential Deep Learning for Open-set Semi-supervised Learning [69.81438976273866]
  Open-set semi-supervised learning (Open-set SSL) considers a more practical scenario, where unlabeled data and test data contain new categories (outliers) not observed in labeled data (inliers) We introduce evidential deep learning (EDL) as an outlier detector to quantify different types of uncertainty, and design different uncertainty metrics for self-training and inference. We propose a novel adaptive negative optimization strategy, making EDL more tailored to the unlabeled dataset containing both inliers and outliers.
  arXiv  Detail & Related papers  (2023-03-21T09:07:15Z)
• Temporal Output Discrepancy for Loss Estimation-based Active Learning [65.93767110342502]
  We present a novel deep active learning approach that queries the oracle for data annotation when the unlabeled sample is believed to incorporate high loss. Our approach achieves superior performances than the state-of-the-art active learning methods on image classification and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2022-12-20T19:29:37Z)
• Pareto Optimization for Active Learning under Out-of-Distribution Data Scenarios [79.02009938011447]
  We propose a sampling scheme, which selects optimal subsets of unlabeled samples with fixed batch size from the unlabeled data pool. Experimental results show its effectiveness on both classical Machine Learning (ML) and Deep Learning (DL) tasks.
  arXiv  Detail & Related papers  (2022-07-04T04:11:44Z)
• Semi-Supervised Active Learning with Temporal Output Discrepancy [42.01906895756629]
  We present a novel deep active learning approach that queries the oracle for data annotation when the unlabeled sample is believed to incorporate high loss. Our approach achieves superior performances than the state-of-the-art active learning methods on image classification and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2021-07-29T16:25:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.