OASIS: Online Sample Selection for Continual Visual Instruction Tuning

• URL: http://arxiv.org/abs/2506.02011v2
• Date: Thu, 09 Oct 2025 08:42:58 GMT
• Title: OASIS: Online Sample Selection for Continual Visual Instruction Tuning
• Authors: Minjae Lee, Minhyuk Seo, Tingyu Qu, Tinne Tuytelaars, Jonghyun Choi,
• Abstract summary: In continual instruction tuning (CIT) scenarios, new instruction tuning data continuously arrive in an online streaming manner.<n>Data selection can mitigate this overhead, but existing strategies often rely on pretrained reference models.<n>Recent reference model-free online sample selection methods address this, but typically select a fixed number of samples per batch.
• Score: 55.92362550389058
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In continual instruction tuning (CIT) scenarios, where new instruction tuning data continuously arrive in an online streaming manner, training delays from large-scale data significantly hinder real-time adaptation. Data selection can mitigate this overhead, but existing strategies often rely on pretrained reference models, which are impractical in CIT setups since future data are unknown. Recent reference model-free online sample selection methods address this, but typically select a fixed number of samples per batch (e.g., top-k), making them vulnerable to distribution shifts where informativeness varies across batches. To address these limitations, we propose OASIS, an adaptive online sample selection approach for CIT that (1) selects informative samples by estimating each sample's informativeness relative to all previously seen data, beyond batch-level constraints, and (2) minimizes informative redundancy of selected samples through iterative selection score updates. Experiments on various large foundation models show that OASIS, using only 25 percent of the data, achieves comparable performance to full-data training and outperforms the state-of-the-art sampling methods.

Related papers

• Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning [49.04912820721943]
  Supervised fine-tuning (SFT) is computationally expensive and sometimes suffers from overfitting or bias amplification.<n>This work studies the online batch selection family that dynamically scores and filters samples during the training process.<n>We develop textbfUDS (Utility-Diversity Sampling), a framework for efficient online batch selection in SFT.
  arXiv  Detail & Related papers  (2025-10-19T15:32:01Z)
• T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning [5.963754140027611]
  Token-Selective HIeRarchical Data Selection for Instruction Tuning (T-SHIRT) is a novel data selection framework.<n>We demonstrate that models instruction-tuned on a curated dataset can outperform those trained on the entire large-scale dataset.
  arXiv  Detail & Related papers  (2025-06-02T04:59:17Z)
• MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuning [69.7347209018861]
  We introduce MLLM-Selector, an automated approach that identifies valuable data for visual instruction tuning.<n>We calculate necessity scores for each sample in the VIT data pool to identify samples pivotal for enhancing model performance.<n>Our findings underscore the importance of mixing necessity and diversity in data choice, leading to the creation of MLLM-Selector.
  arXiv  Detail & Related papers  (2025-03-26T12:42:37Z)
• Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigm [50.492124556982674]
  This paper introduces a novel choice-based sample selection framework.<n>It shifts the focus from evaluating individual sample quality to comparing the contribution value of different samples.<n>We validate our approach on a larger medical dataset, highlighting its practical applicability in real-world applications.
  arXiv  Detail & Related papers  (2025-03-04T07:32:41Z)
• A CLIP-Powered Framework for Robust and Generalizable Data Selection [51.46695086779598]
  Real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance.<n>Data selection has shown promise in identifying the most representative samples from the entire dataset.<n>We propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection.
  arXiv  Detail & Related papers  (2024-10-15T03:00:58Z)
• Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection [89.42023974249122]
  Adapt-$infty$ is a new multi-way and adaptive data selection approach for lifelong instruction tuning.<n>We construct pseudo-skill clusters by grouping gradient-based sample vectors.<n>We select the best-performing data selector for each skill cluster from a pool of selector experts.<n>This data selector samples a subset of the most important samples from each skill cluster for training.
  arXiv  Detail & Related papers  (2024-10-14T15:48:09Z)
• Diversified Batch Selection for Training Acceleration [68.67164304377732]
  A prevalent research line, known as online batch selection, explores selecting informative subsets during the training process. vanilla reference-model-free methods involve independently scoring and selecting data in a sample-wise manner. We propose Diversified Batch Selection (DivBS), which is reference-model-free and can efficiently select diverse and representative samples.
  arXiv  Detail & Related papers  (2024-06-07T12:12:20Z)
• Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection [59.11430077029321]
  We introduce a novel dataset selection method, Self-Filter, for vision-language models (VLMs) In the first stage, we devise a scoring network to evaluate the difficulty of training instructions, which is co-trained with the VLM. In the second stage, we use the trained score net to measure the difficulty of each instruction, select the most challenging samples, and penalize similar samples to encourage diversity.
  arXiv  Detail & Related papers  (2024-02-19T20:08:48Z)
• Towards Free Data Selection with General-Purpose Models [71.92151210413374]
  A desirable data selection algorithm can efficiently choose the most informative samples to maximize the utility of limited annotation budgets. Current approaches, represented by active learning methods, typically follow a cumbersome pipeline that iterates the time-consuming model training and batch data selection repeatedly. FreeSel bypasses the heavy batch selection process, achieving a significant improvement in efficiency and being 530x faster than existing active learning methods.
  arXiv  Detail & Related papers  (2023-09-29T15:50:14Z)
• AdaSelection: Accelerating Deep Learning Training through Data Subsampling [27.46630703428186]
  We introduce AdaSelection, an adaptive sub-sampling method to identify the most informative sub-samples within each minibatch. Compared with industry-standard baselines, AdaSelection consistently displays superior performance.
  arXiv  Detail & Related papers  (2023-06-19T07:01:28Z)
• A Gradient-based Approach for Online Robust Deep Neural Network Training with Noisy Labels [27.7867122240632]
  In this paper, we propose a novel-based approach to enable the online selection of noisy labels. Online Gradient-based Selection Selection (OGRS) can automatically select clean samples by steps of update from datasets with varying clean ratios without changing the parameter setting.
  arXiv  Detail & Related papers  (2023-06-08T08:57:06Z)
• Classifier Transfer with Data Selection Strategies for Online Support Vector Machine Classification with Class Imbalance [1.2599533416395767]
  We focus on data selection strategies which limit the size of the stored training data. We show that by using the right combination of data selection criteria, it is possible to adapt the classifier and largely increase the performance.
  arXiv  Detail & Related papers  (2022-08-10T02:36:20Z)
• CAFA: Class-Aware Feature Alignment for Test-Time Adaptation [50.26963784271912]
  Test-time adaptation (TTA) aims to address this challenge by adapting a model to unlabeled data at test time. We propose a simple yet effective feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which simultaneously encourages a model to learn target representations in a class-discriminative manner.
  arXiv  Detail & Related papers  (2022-06-01T03:02:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.