Meta-Sel: Efficient Demonstration Selection for In-Context Learning via Supervised Meta-Learning

• URL: http://arxiv.org/abs/2602.12123v1
• Date: Thu, 12 Feb 2026 16:11:29 GMT
• Title: Meta-Sel: Efficient Demonstration Selection for In-Context Learning via Supervised Meta-Learning
• Authors: Xubin Wang, Weijia Jia,
• Abstract summary: We propose Meta-Sel, a lightweight supervised meta-learning approach for intent classification.<n>It learns a fast, interpretable scoring function for (candidate, query) pairs from labeled training data.<n>At inference time, the selector performs a single vectorized scoring over the full candidate pool and returns the top-k demonstrations.
• Score: 9.851186633544975
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Demonstration selection is a practical bottleneck in in-context learning (ICL): under a tight prompt budget, accuracy can change substantially depending on which few-shot examples are included, yet selection must remain cheap enough to run per query over large candidate pools. We propose Meta-Sel, a lightweight supervised meta-learning approach for intent classification that learns a fast, interpretable scoring function for (candidate, query) pairs from labeled training data. Meta-Sel constructs a meta-dataset by sampling pairs from the training split and using class agreement as supervision, then trains a calibrated logistic regressor on two inexpensive meta-features: TF--IDF cosine similarity and a length-compatibility ratio. At inference time, the selector performs a single vectorized scoring pass over the full candidate pool and returns the top-k demonstrations, requiring no model fine-tuning, no online exploration, and no additional LLM calls. This yields deterministic rankings and makes the selection mechanism straightforward to audit via interpretable feature weights. Beyond proposing Meta-Sel, we provide a broad empirical study of demonstration selection, benchmarking 12 methods -- spanning prompt engineering baselines, heuristic selection, reinforcement learning, and influence-based approaches -- across four intent datasets and five open-source LLMs. Across this benchmark, Meta-Sel consistently ranks among the top-performing methods, is particularly effective for smaller models where selection quality can partially compensate for limited model capacity, and maintains competitive selection-time overhead.

Related papers

• MetaRank: Task-Aware Metric Selection for Model Transferability Estimation [25.11923464702804]
  We introduce MetaRank, a meta-learning framework for automatic, task-aware MTE metric selection.<n>Rather than relying on conventional meta-features, MetaRank encodes textual descriptions of both datasets and MTE metrics.<n>A meta-predictor is then trained offline on diverse meta-tasks to learn the intricate relationship between dataset characteristics and metric mechanisms.
  arXiv  Detail & Related papers  (2025-11-26T03:15:13Z)
• Imbalanced Regression Pipeline Recommendation [5.863538874435322]
  This work proposes the Meta-learning for Imbalanced Regression (Meta-IR) framework.<n>It trains meta-classifiers to recommend the best pipeline composed of the resampling strategy and learning model per task.<n>Compared with AutoML frameworks, Meta-IR obtained better results.
  arXiv  Detail & Related papers  (2025-07-16T04:34:02Z)
• Large Language Models are Demonstration Pre-Selectors for Themselves [57.101804269100185]
  In-context learning (ICL) with large language models (LLMs) delivers strong few-shot performance by choosing few-shot demonstrations from the entire training data.<n>FEw yet Essential Demonstration prE-selectoR is a novel pre-selection framework that identifies a representative subset of demonstrations.<n>FEw yet Essential Demonstration prE-selectoR can reduce training data size by over 20% while maintaining performance.
  arXiv  Detail & Related papers  (2025-06-06T12:29:03Z)
• MLRS-PDS: A Meta-learning recommendation of dynamic ensemble selection pipelines [3.1140073169854485]
  This paper introduces a meta-learning recommendation system (MLRS) to recommend the optimal pool generation scheme for dynamic ensemble selection. The system employs a meta-model built from dataset meta-features to predict the most suitable pool generation scheme and DES method for a given dataset. We demonstrate that this meta-learning recommendation system outperforms traditional fixed pool or DES method selection strategies.
  arXiv  Detail & Related papers  (2024-07-10T10:31:57Z)
• LLM-Select: Feature Selection with Large Language Models [64.5099482021597]
  Large language models (LLMs) are capable of selecting the most predictive features, with performance rivaling the standard tools of data science.<n>Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place.
  arXiv  Detail & Related papers  (2024-07-02T22:23:40Z)
• Aligning Language Models with Demonstrated Feedback [58.834937450242975]
  Demonstration ITerated Task Optimization (DITTO) directly aligns language model outputs to a user's demonstrated behaviors.<n>We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts.
  arXiv  Detail & Related papers  (2024-06-02T23:13:56Z)
• Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection [59.11430077029321]
  We introduce a novel dataset selection method, Self-Filter, for vision-language models (VLMs) In the first stage, we devise a scoring network to evaluate the difficulty of training instructions, which is co-trained with the VLM. In the second stage, we use the trained score net to measure the difficulty of each instruction, select the most challenging samples, and penalize similar samples to encourage diversity.
  arXiv  Detail & Related papers  (2024-02-19T20:08:48Z)
• MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuning [68.12870241637636]
  We propose MILO, a model-agnostic subset selection framework that decouples the subset selection from model training. Our empirical results indicate that MILO can train models $3times - 10 times$ faster and tune hyperparameters $20times - 75 times$ faster than full-dataset training or tuning without performance.
  arXiv  Detail & Related papers  (2023-01-30T20:59:30Z)
• Meta-Generating Deep Attentive Metric for Few-shot Classification [53.07108067253006]
  We present a novel deep metric meta-generation method to generate a specific metric for a new few-shot learning task. In this study, we structure the metric using a three-layer deep attentive network that is flexible enough to produce a discriminative metric for each task. We gain surprisingly obvious performance improvement over state-of-the-art competitors, especially in the challenging cases.
  arXiv  Detail & Related papers  (2020-12-03T02:07:43Z)
• A Markov Decision Process Approach to Active Meta Learning [24.50189361694407]
  In supervised learning, we fit a single statistical model to a given data set, assuming that the data is associated with a singular task. In meta-learning, the data is associated with numerous tasks, and we seek a model that may perform well on all tasks simultaneously.
  arXiv  Detail & Related papers  (2020-09-10T15:45:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.