Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation

• URL: http://arxiv.org/abs/2603.19264v1
• Date: Thu, 26 Feb 2026 22:03:51 GMT
• Title: Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation
• Abstract summary: Existing frameworks for active sample selection offer limited support for generative Question Answering tasks.<n>We present Generative Active Testing (GAT), an uncertainty-aware acquisition framework leveraging Large Language Models (LLM) as surrogates for informing the sample selection process.<n>Our zero-shot acquisition functions reduce estimation error by 40% compared to traditional sampling baselines, offering a scalable solution for cost-effective model benchmarking.
• Score: 6.370860526262166
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the widespread adoption of pre-trained Large Language Models (LLM), there exists a high demand for task-specific test sets to benchmark their performance in domains such as healthcare and biomedicine. However, the cost of labeling test samples while developing new benchmarks poses a significant challenge, especially when expert annotators are required. Existing frameworks for active sample selection offer limited support for generative Question Answering tasks, where option dynamics can affect model decision boundaries. In this paper, we present Generative Active Testing (GAT), an uncertainty-aware acquisition framework leveraging LLMs as surrogates for informing the sample selection process. Using a novel Statement Adaptation Module, we modify generative tasks into a pseudo-classification format, enabling the capture of sample-level uncertainties across unlabeled candidates. Our zero-shot acquisition functions reduce estimation error by ~40% compared to traditional sampling baselines, offering a scalable solution for cost-effective model benchmarking.

Related papers

• Select, Label, Evaluate: Active Testing in NLP [13.476392417672812]
  Active Testing is a framework that selects the most informative test samples for annotation.<n>Given a labeling budget, it aims to choose the subset that best estimates model performance.<n>Experiments show annotation reductions of up to 95%, with performance estimation accuracy difference from the full test set within 1%.
  arXiv  Detail & Related papers  (2026-03-23T11:28:57Z)
• Efficient Evaluation of LLM Performance with Statistical Guarantees [11.703733256169214]
  We propose Factorized Active Querying (FAQ) for benchmarking large language models.<n>FAQ adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy.<n>FAQ delivers up to $5times$ effective sample size gains over strong baselines on two benchmark suites.
  arXiv  Detail & Related papers  (2026-01-28T04:59:20Z)
• Table Detection with Active Learning [1.9881456274482427]
  Active learning is a promising solution to minimize annotation costs by selecting the most informative samples.<n>Our approach ensures the selection of representative examples that improve model generalization.<n>Our results demonstrate that AL-based example selection significantly outperforms random sampling.
  arXiv  Detail & Related papers  (2025-09-24T11:22:30Z)
• Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigm [50.492124556982674]
  This paper introduces a novel choice-based sample selection framework.<n>It shifts the focus from evaluating individual sample quality to comparing the contribution value of different samples.<n>We validate our approach on a larger medical dataset, highlighting its practical applicability in real-world applications.
  arXiv  Detail & Related papers  (2025-03-04T07:32:41Z)
• On Speeding Up Language Model Evaluation [48.51924035873411]
  We propose an $textitadaptive$ approach to explore this space.<n>We lean on multi-armed bandits to sequentially identify the next (method, validation sample)-pair to evaluate.<n>We show that it can identify the top-performing method using only 5-15% of the typical resources.
  arXiv  Detail & Related papers  (2024-07-08T17:48:42Z)
• Diversified Batch Selection for Training Acceleration [68.67164304377732]
  A prevalent research line, known as online batch selection, explores selecting informative subsets during the training process. vanilla reference-model-free methods involve independently scoring and selecting data in a sample-wise manner. We propose Diversified Batch Selection (DivBS), which is reference-model-free and can efficiently select diverse and representative samples.
  arXiv  Detail & Related papers  (2024-06-07T12:12:20Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models [55.01592097059969]
  Supervised finetuning on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool. We propose using experimental design to circumvent the computational bottlenecks of active learning.
  arXiv  Detail & Related papers  (2024-01-12T16:56:54Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)
• On the Limits of Evaluating Embodied Agent Model Generalization Using Validation Sets [101.28658250723804]
  This paper experiments with augmenting a transformer model with modules that effectively utilize a wider field of view and learn to choose whether the next step requires a navigation or manipulation action. We observe that the proposed modules resulted in improved, and in fact state-of-the-art performance on an unseen validation set of a popular benchmark dataset, ALFRED. We highlight this result as we believe it may be a wider phenomenon in machine learning tasks but primarily noticeable only in benchmarks that limit evaluations on test splits.
  arXiv  Detail & Related papers  (2022-05-18T23:52:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.