Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training

• URL: http://arxiv.org/abs/2403.03728v1
• Date: Wed, 6 Mar 2024 14:18:24 GMT
• Title: Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training
• Authors: Paul Doucet, Benjamin Estermann, Till Aczel, Roger Wattenhofer
• Abstract summary: This study addresses the integration of diversity-based and uncertainty-based sampling strategies in active learning. We introduce a straightforward called TCM that mitigates the cold start problem while maintaining strong performance across various data levels.
• Score: 23.573986817769025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study addresses the integration of diversity-based and uncertainty-based sampling strategies in active learning, particularly within the context of self-supervised pre-trained models. We introduce a straightforward heuristic called TCM that mitigates the cold start problem while maintaining strong performance across various data levels. By initially applying TypiClust for diversity sampling and subsequently transitioning to uncertainty sampling with Margin, our approach effectively combines the strengths of both strategies. Our experiments demonstrate that TCM consistently outperforms existing methods across various datasets in both low and high data regimes.

Related papers

• On-the-fly Modulation for Balanced Multimodal Learning [53.616094855778954]
  Multimodal learning is expected to boost model performance by integrating information from different modalities. The widely-used joint training strategy leads to imbalanced and under-optimized uni-modal representations. We propose On-the-fly Prediction Modulation (OPM) and On-the-fly Gradient Modulation (OGM) strategies to modulate the optimization of each modality.
  arXiv  Detail & Related papers  (2024-10-15T13:15:50Z)
• Cross-Modal Few-Shot Learning: a Generative Transfer Learning Framework [58.362064122489166]
  This paper introduces the Cross-modal Few-Shot Learning task, which aims to recognize instances from multiple modalities when only a few labeled examples are available. We propose a Generative Transfer Learning framework consisting of two stages: the first involves training on abundant unimodal data, and the second focuses on transfer learning to adapt to novel data. Our finds demonstrate that GTL has superior performance compared to state-of-the-art methods across four distinct multi-modal datasets.
  arXiv  Detail & Related papers  (2024-10-14T16:09:38Z)
• MITA: Bridging the Gap between Model and Data for Test-time Adaptation [68.62509948690698]
  Test-Time Adaptation (TTA) has emerged as a promising paradigm for enhancing the generalizability of models. We propose Meet-In-The-Middle based MITA, which introduces energy-based optimization to encourage mutual adaptation of the model and data from opposing directions.
  arXiv  Detail & Related papers  (2024-10-12T07:02:33Z)
• On Task Performance and Model Calibration with Supervised and Self-Ensembled In-Context Learning [71.44986275228747]
  In-context learning (ICL) has become an efficient approach propelled by the recent advancements in large language models (LLMs) However, both paradigms are prone to suffer from the critical problem of overconfidence (i.e., miscalibration)
  arXiv  Detail & Related papers  (2023-12-21T11:55:10Z)
• Self-aware and Cross-sample Prototypical Learning for Semi-supervised Medical Image Segmentation [10.18427897663732]
  Consistency learning plays a crucial role in semi-supervised medical image segmentation. It enables the effective utilization of limited annotated data while leveraging the abundance of unannotated data. We propose a self-aware and cross-sample prototypical learning method ( SCP-Net) to enhance the diversity of prediction in consistency learning.
  arXiv  Detail & Related papers  (2023-05-25T16:22:04Z)
• On Modality Bias Recognition and Reduction [70.69194431713825]
  We study the modality bias problem in the context of multi-modal classification. We propose a plug-and-play loss function method, whereby the feature space for each label is adaptively learned. Our method yields remarkable performance improvements compared with the baselines.
  arXiv  Detail & Related papers  (2022-02-25T13:47:09Z)
• Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions [91.63716984911278]
  We introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adaptive integration of different modalities and produces a trustworthy regression result. Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multimodal regression tasks.
  arXiv  Detail & Related papers  (2021-11-11T14:28:12Z)
• Mitigating Sampling Bias and Improving Robustness in Active Learning [13.994967246046008]
  We introduce supervised contrastive active learning by leveraging the contrastive loss for active learning under a supervised setting. We propose an unbiased query strategy that selects informative data samples of diverse feature representations. We empirically demonstrate our proposed methods reduce sampling bias, achieve state-of-the-art accuracy and model calibration in an active learning setup.
  arXiv  Detail & Related papers  (2021-09-13T20:58:40Z)
• Deep Active Learning with Augmentation-based Consistency Estimation [23.492616938184092]
  We propose a methodology to improve generalization ability, by applying data augmentation-based techniques to an active learning scenario. For the data augmentation-based regularization loss, we redefined cutout (co) and cutmix (cm) strategies as quantitative metrics. We have shown that the augmentation-based regularizer can lead to improved performance on the training step of active learning.
  arXiv  Detail & Related papers  (2020-11-05T05:22:58Z)
• Ask-n-Learn: Active Learning via Reliable Gradient Representations for Image Classification [29.43017692274488]
  Deep predictive models rely on human supervision in the form of labeled training data. We propose Ask-n-Learn, an active learning approach based on gradient embeddings obtained using the pesudo-labels estimated in each of the algorithm.
  arXiv  Detail & Related papers  (2020-09-30T05:19:56Z)
• Progressive Multi-Stage Learning for Discriminative Tracking [25.94944743206374]
  We propose a joint discriminative learning scheme with the progressive multi-stage optimization policy of sample selection for robust visual tracking. The proposed scheme presents a novel time-weighted and detection-guided self-paced learning strategy for easy-to-hard sample selection. Experiments on the benchmark datasets demonstrate the effectiveness of the proposed learning framework.
  arXiv  Detail & Related papers  (2020-04-01T07:01:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.