Zero-Shot Active Feature Acquisition via LLM-Elicitation

• URL: http://arxiv.org/abs/2606.18933v2
• Date: Thu, 18 Jun 2026 07:16:30 GMT
• Title: Zero-Shot Active Feature Acquisition via LLM-Elicitation
• Abstract summary: Large language models (LLMs) supply unsupervised domain knowledge, but are poor sequential planners.<n>We develop a framework for zero-shot AFA through disciplined elicitation.<n>We apply our framework to two settings: binary classification and top-$k$ identification.
• Score: 33.792645413345845
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Active feature acquisition (AFA) sequentially selects which features to observe to reach a classification or ranking decision. Its central limitation is reliance on large amount of labeled data to fit probabilistic models guiding acquisition. Large language models (LLMs) supply unsupervised domain knowledge, but are poor sequential planners. Asking one to both know and decide conflates capabilities best kept separate. Here, we develop a framework for zero-shot AFA through disciplined elicitation: asking the LLM only for what it can be trusted to return, the unary deviations and pairwise co-variations that are the sufficient statistics of a Markov random field (MRF). We apply our framework to two settings: binary classification and top-$k$ identification. In practice, the LLM reliably returns only discriminative statistics, what distinguishes the classes rather than each class in isolation, which precludes classical AFA. We apply a maximum-entropy closure that resolves this gauge ambiguity. We evaluate on a cohort of Inflammatory Bowel Disease (IBD) patients, an active clinical setting where diagnostic ambiguity and patient heterogeneity obstruct stable treatment strategies. Our framework outperforms the LLM both on real labels and on its own extracted beliefs. Where it matters most, on the hardest patients, our top-$k$ acquisition policy markedly outperforms all existing methods.

Related papers

• Can Revealed Preferences Clarify LLM Alignment and Steering? [23.175180848107093]
  We present an empirical pipeline for estimating the implied preferences that an LLM's observed choices optimize.<n>We show how this revealed-preference description allows rigorous evaluation of whether models behave in a consistently goal-directed way.
  arXiv  Detail & Related papers  (2026-05-08T23:26:35Z)
• Synergizing Discriminative Exemplars and Self-Refined Experience for MLLM-based In-Context Learning in Medical Diagnosis [4.619078510367921]
  Clinician Mimetic is a novel In-Context Learning (ICL) framework designed to synergize Discriminative Exemplar Coreset Selection (DECS) and Self-Refined Experience Summarization (SRES)<n> DECS simulates a clinician's ability to reference "anchor cases" by selecting discriminative visual coresets from noisy data at the computational level.<n>SRES mimics the cognition and reflection in clinical diagnosis by distilling diverse rollouts into a dynamic textual Experience Bank.
  arXiv  Detail & Related papers  (2026-03-29T15:29:59Z)
• Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools [0.17188280334580197]
  Reranking evaluations study how a reranker orders candidates returned by an upstream retriever.<n>This setup couples ranking behavior with retrieval quality, so differences in output cannot be attributed to the ranking policy alone.<n>We introduce a controlled diagnostic that isolates reranking by using Multi-News clusters as fixed evidence pools.
  arXiv  Detail & Related papers  (2026-02-20T21:07:32Z)
• Do LLMs Act Like Rational Agents? Measuring Belief Coherence in Probabilistic Decision Making [28.256934953904317]
  We study whether large language models (LLMs) are rational utility maximizers with coherent beliefs and stable preferences.<n>Our approach provides falsifiable conditions under which the reported probabilities emphcannot correspond to the true beliefs of any rational agent.
  arXiv  Detail & Related papers  (2026-02-06T00:50:33Z)
• Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models [55.94503936470247]
  Large-scale AI evaluation increasingly relies on aggregating binary judgments from $K$ annotators, including judges.<n>Most classical methods assume annotators are conditionally independent given the true label $Yin0,1$, an assumption often violated by LLM judges.<n>We study label aggregation through a hierarchy of dependence-aware models based on Ising graphical models and latent factors.
  arXiv  Detail & Related papers  (2026-01-29T21:26:50Z)
• Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLM [3.599447860592914]
  We introduce the Frequency-Based Probabilistic Ranker (FBPR)<n>FBPR scores options with a smoothed Naive Bayes over concept-diagnosis co-occurrence statistics from a large corpus.<n>We show that an approach similar to the historically grounded, low-complexity expert systems still accounts for a substantial portion of benchmark performance.
  arXiv  Detail & Related papers  (2025-12-14T23:00:10Z)
• Self-ensemble: Mitigating Confidence Mis-calibration for Large Language Models [67.62810111789338]
  Large Language Models exhibit a confidence distortion problem on multi-choice question-answering.<n>We propose Self-ensemble to solve this problem.<n> Experimental results on three LLMs and datasets demonstrate that Self-ensemble comprehensively addresses the confidence distortion problem.
  arXiv  Detail & Related papers  (2025-06-02T17:59:29Z)
• Ranked from Within: Ranking Large Multimodal Models Without Labels [73.96543593298426]
  We show that uncertainty scores derived from softmax distributions provide a robust basis for ranking models across various tasks.<n>This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.
  arXiv  Detail & Related papers  (2024-12-09T13:05:43Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data [102.16105233826917]
  Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised learning, on-policy reinforcement learning (RL), and contrastive learning.
  arXiv  Detail & Related papers  (2024-04-22T17:20:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.