Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation

• URL: http://arxiv.org/abs/2607.05891v1
• Date: Tue, 07 Jul 2026 06:40:58 GMT
• Title: Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation
• Abstract summary: Few-medoids is an embarrassingly simple coreset selection strategy that chooses the samples closest to the centroid of each class.<n>We present extensive KD experiments on four datasets, covering a wide range of image classification problems.<n>Our empirical results indicate that few-medoids is able to consistently surpass the random selection baseline.
• Score: 26.200974332824966
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Coreset selection aims to identify a small and highly representative subset of a massive dataset for efficient model training. The problem remains challenging even in the few-shot knowledge distillation (KD) setup, where a full-scale pre-trained teacher informs the student network. Typical sample selection strategies often struggle to surpass the random selection baseline. In this paper, we showcase few-medoids, an embarrassingly simple coreset selection strategy that chooses the samples closest to the centroid (average image) of each class. We present extensive KD experiments on four datasets, covering a wide range of image classification problems, and three teacher-student model pairs, comprising both convolutional and transformer networks. Although the proposed method is embarrassingly simple, our empirical results indicate that few-medoids is able to consistently surpass the random selection baseline, as well as the other coreset selection strategies. We therefore consider that few-medoids can be used as a drop-in replacement for commonly-used baselines (e.g. herding or k-center Greedy), in future research on coreset selection. To reproduce the reported results, we publicly release our code at https://github.com/CemilAndreiDilmac/Few-Shot-KD-Coreset.

Related papers

• A Coreset Selection Framework with Ensemble Aggregation for Image Classification [3.122537594115402]
  We propose a framework that combines coreset selection with an ensemble aggregation over multiple runs.<n>For coreset selection, we propose SCOre-Stratified Selection (SCOSS), which partitions the training data into intervals based on a chosen score and samples from each interval.<n>The ensemble combines predictions from multiple runs, each performed on an independently sampled training subset.
  arXiv  Detail & Related papers  (2026-07-10T05:09:12Z)
• MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection [59.34281845159177]
  We propose MIRA, a source-aware filtering framework based on self-anchored rubric discovery.<n>On code-oriented mid-training with 21 sources and 5 source groups, MIRA outperforms selection baselines across nine code benchmarks.
  arXiv  Detail & Related papers  (2026-05-28T17:40:40Z)
• Class Balance Matters to Active Class-Incremental Learning [61.11786214164405]
  We aim to start from a pool of large-scale unlabeled data and then annotate the most informative samples for incremental learning.<n>We propose Class-Balanced Selection (CBS) strategy to achieve both class balance and informativeness in chosen samples.<n>Our CBS can be plugged and played into those CIL methods which are based on pretrained models with prompts tunning technique.
  arXiv  Detail & Related papers  (2024-12-09T16:37:27Z)
• Zero-Shot Coreset Selection: Efficient Pruning for Unlabeled Data [22.45812577928658]
  Coreset selection aims to find a representative subset of data to train models. ZCore is a method that efficiently selects coresets without ground truth labels or training on candidate data. We evaluate ZCore on four datasets and outperform several state-of-the-art label-based methods.
  arXiv  Detail & Related papers  (2024-11-22T21:17:49Z)
• Diverse Subset Selection via Norm-Based Sampling and Orthogonality [31.558151874765667]
  Large annotated datasets are crucial for the success of deep neural networks, but labeling data can be prohibitively expensive in domains such as medical imaging.<n>This work tackles the subset selection problem: selecting a small set of the most informative examples from a large unlabeled pool for annotation.
  arXiv  Detail & Related papers  (2024-06-03T08:12:32Z)
• Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints [69.27190330994635]
  Coreset selection is powerful in reducing computational costs and accelerating data processing for deep learning algorithms. We propose an innovative method, which maintains optimization priority order over the model performance and coreset size. Empirically, extensive experiments confirm its superiority, often yielding better model performance with smaller coreset sizes.
  arXiv  Detail & Related papers  (2023-11-15T03:43:04Z)
• D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning [70.98091101459421]
  Coreset selection seeks to select a subset of the training data so as to maximize the performance of models trained on this subset, also referred to as coreset. We propose a novel pruning algorithm, D2 Pruning, that uses forward and reverse message passing over this dataset graph for coreset selection. Results show that D2 Pruning improves coreset selection over previous state-of-the-art methods for up to 70% pruning rates.
  arXiv  Detail & Related papers  (2023-10-11T23:01:29Z)
• Which Pixel to Annotate: a Label-Efficient Nuclei Segmentation Framework [70.18084425770091]
  Deep neural networks have been widely applied in nuclei instance segmentation of H&E stained pathology images. It is inefficient and unnecessary to label all pixels for a dataset of nuclei images which usually contain similar and redundant patterns. We propose a novel full nuclei segmentation framework that chooses only a few image patches to be annotated, augments the training set from the selected samples, and achieves nuclei segmentation in a semi-supervised manner.
  arXiv  Detail & Related papers  (2022-12-20T14:53:26Z)
• Coverage-centric Coreset Selection for High Pruning Rates [11.18635356469467]
  One-shot coreset selection aims to select a subset of the training data, given a pruning rate, that can achieve high accuracy for models that are subsequently trained only with that subset. State-of-the-art coreset selection methods typically assign an importance score to each example and select the most important examples to form a coreset. But at high pruning rates, they have been found to suffer a catastrophic accuracy drop, performing worse than even random coreset selection.
  arXiv  Detail & Related papers  (2022-10-28T00:14:00Z)
• DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning [3.897574108827803]
  We provide an empirical study on popular coreset selection methods on CIFAR10 and ImageNet datasets. Although some methods perform better in certain experiment settings, random selection is still a strong baseline.
  arXiv  Detail & Related papers  (2022-04-18T18:14:30Z)
• Online Coreset Selection for Rehearsal-based Continual Learning [65.85595842458882]
  In continual learning, we store a subset of training examples (coreset) to be replayed later to alleviate catastrophic forgetting. We propose Online Coreset Selection (OCS), a simple yet effective method that selects the most representative and informative coreset at each iteration. Our proposed method maximizes the model's adaptation to a target dataset while selecting high-affinity samples to past tasks, which directly inhibits catastrophic forgetting.
  arXiv  Detail & Related papers  (2021-06-02T11:39:25Z)
• Extending Contrastive Learning to Unsupervised Coreset Selection [26.966136750754732]
  We propose an unsupervised way of selecting a core-set entirely unlabeled. We use two leading methods for contrastive learning. Compared with existing coreset selection methods with labels, our approach reduced the cost associated with human annotation.
  arXiv  Detail & Related papers  (2021-03-05T10:21:51Z)
• Attentive CutMix: An Enhanced Data Augmentation Approach for Deep Learning Based Image Classification [58.20132466198622]
  We propose Attentive CutMix, a naturally enhanced augmentation strategy based on CutMix. In each training iteration, we choose the most descriptive regions based on the intermediate attention maps from a feature extractor. Our proposed method is simple yet effective, easy to implement and can boost the baseline significantly.
  arXiv  Detail & Related papers  (2020-03-29T15:01:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.