LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection

• URL: http://arxiv.org/abs/2510.26510v2
• Date: Thu, 06 Nov 2025 09:42:34 GMT
• Title: LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection
• Authors: Youssef Attia El Hili, Albert Thomas, Malik Tiomoko, Abdelhakim Benechehab, Corentin Léger, Corinne Ancourt, Balázs Kégl,
• Abstract summary: We investigate whether large language models (LLMs) can act as in-context meta-learners.<n>We show that LLMs can exploit dataset metadata to recommend competitive models and hyperparameters without search.<n>These results highlight a promising new role for LLMs as lightweight, general-purpose assistants.
• Score: 8.94883745636267
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model and hyperparameter selection are critical but challenging in machine learning, typically requiring expert intuition or expensive automated search. We investigate whether large language models (LLMs) can act as in-context meta-learners for this task. By converting each dataset into interpretable metadata, we prompt an LLM to recommend both model families and hyperparameters. We study two prompting strategies: (1) a zero-shot mode relying solely on pretrained knowledge, and (2) a meta-informed mode augmented with examples of models and their performance on past tasks. Across synthetic and real-world benchmarks, we show that LLMs can exploit dataset metadata to recommend competitive models and hyperparameters without search, and that improvements from meta-informed prompting demonstrate their capacity for in-context meta-learning. These results highlight a promising new role for LLMs as lightweight, general-purpose assistants for model selection and hyperparameter optimization.

Related papers

• Can LLM Annotations Replace User Clicks for Learning to Rank? [112.2254432364736]
  Large-scale supervised data is essential for training modern ranking models, but obtaining high-quality human annotations is costly.<n>Click data has been widely used as a low-cost alternative, and with recent advances in large language models (LLMs), LLM-based relevance annotation has emerged as another promising annotation.<n> Experiments on both a public dataset, TianGong-ST, and an industrial dataset, Baidu-Click, show that click-supervised models perform better on high-frequency queries.<n>We explore two training strategies -- data scheduling and frequency-aware multi-objective learning -- that integrate both supervision signals.
  arXiv  Detail & Related papers  (2025-11-10T02:26:14Z)
• DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression [7.1654056866441245]
  Large language models (LLMs) excel in general tasks but struggle with domain-specific ones, requiring fine-tuning with specific data.<n>We introduce a Data and Model Compression Framework (DaMoC) that addresses this challenge.<n>We show that we can select the optimal LLM while saving approximately 20-fold in training time.
  arXiv  Detail & Related papers  (2025-09-01T08:06:49Z)
• Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models [7.075648770762989]
  Fine-tuning large language models with low-rank adaptaion (LoRA) is a cost-effective way to incorporate information from a specific dataset.<n>It is often unclear how well the fine-tuned LLM will generalize, i.e., how well it will perform on unseen datasets.<n>We propose Amortized Bayesian Meta-Learning for LoRA (ABMLL) to improve generalization and scales to large models.
  arXiv  Detail & Related papers  (2025-08-19T21:57:59Z)
• Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques [14.892995952768352]
  Language Models (LMs) have excelled at tasks like text generation, summarization, and question answering.<n>Their inference remains computationally expensive and energy intensive in settings with limited hardware, power, or bandwidth.<n>Recent approaches have introduced multi LLM intelligent model selection strategies that dynamically allocate computational resources based on query complexity.
  arXiv  Detail & Related papers  (2025-06-06T23:13:08Z)
• Efficient Model Selection for Time Series Forecasting via LLMs [52.31535714387368]
  We propose to leverage Large Language Models (LLMs) as a lightweight alternative for model selection.<n>Our method eliminates the need for explicit performance matrices by utilizing the inherent knowledge and reasoning capabilities of LLMs.
  arXiv  Detail & Related papers  (2025-04-02T20:33:27Z)
• The Performance of the LSTM-based Code Generated by Large Language Models (LLMs) in Forecasting Time Series Data [0.3749861135832072]
  This paper investigates and compares the performance of the mainstream LLMs, such as ChatGPT, PaLM, LLama, and Falcon, in generating deep learning models for analyzing time series data.<n>The results can be beneficial for data analysts and practitioners who would like to leverage generative AIs to produce good prediction models with acceptable goodness.
  arXiv  Detail & Related papers  (2024-11-27T20:18:36Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• Verbalized Machine Learning: Revisiting Machine Learning with Language Models [63.10391314749408]
  We introduce the framework of verbalized machine learning (VML)<n>VML constrains the parameter space to be human-interpretable natural language.<n>We empirically verify the effectiveness of VML, and hope that VML can serve as a stepping stone to stronger interpretability.
  arXiv  Detail & Related papers  (2024-06-06T17:59:56Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs)<n>We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning [43.512739869120125]
  We propose MAML-en-LLM, a novel method for meta-training large language models (LLMs) MAML-en-LLM can learn truly generalizable parameters that not only perform well on disjointed tasks but also adapts to unseen tasks. We demonstrate that MAML-en-LLM outperforms baselines in settings with limited amount of training data on both seen and unseen domains.
  arXiv  Detail & Related papers  (2024-05-19T04:49:42Z)
• Entity Matching using Large Language Models [3.7277730514654555]
  This paper investigates using generative large language models (LLMs) as a less task-specific training data-dependent alternative to PLM-based matchers. We show that GPT4 can generate structured explanations for matching decisions and can automatically identify potential causes of matching errors.
  arXiv  Detail & Related papers  (2023-10-17T13:12:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.