Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking

• URL: http://arxiv.org/abs/2608.14808v1
• Date: Fri, 14 Aug 2026 18:23:32 GMT
• Title: Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking
• Abstract summary: We formalize multi-turn information seeking as solving a k-underspecified constraint satisfaction problem.<n>We evaluate models along three axes: what they ask, when they ask it, and how the acquired information affects the final answer.
• Score: 27.6486449009769
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When a user question is underspecified, a capable model should recognize that its context is insufficient, identify the missing information, ask for it, and respond only once that information determines a unique answer. We formalize multi-turn information seeking as solving a k-underspecified constraint satisfaction problem, where k is the number of variables jointly required to determine the target and therefore measures the degree of missing information. We instantiate the formulation in MT-InfoSeek, a controlled evaluation suite of 5,251 problems and 9,006 task instances spanning mathematics, logic, biology, medicine, and general knowledge. We evaluate models along three axes: what they ask, when they ask it, and how the acquired information affects the final answer. Performance degrades across models and domains as underspecification increases. Models recognize that additional information is needed but underestimate how much, and in logical problems at k = 2 they under-predict the degree of missing information about four times as often as they over-predict it. They also fail to identify a minimal sufficient set of queries, improve only marginally when given the true k, and often stop before acquiring sufficient information. In tasks with ordered dependencies, an incorrect query order reduces final accuracy even when the model eventually acquires all necessary information. We measure information seeking directly through final sufficiency, which records whether the acquired information determines the target independent of answer generation. This separation shows differences between models that final accuracy alone does not capture, and indicates that the ability to seek information over multiple turns is distinct from the ability to generate answers and is not measured by current LLM evaluations.

Related papers

• MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning [0.0]
  MIRA-Math is a benchmark for solving problems whose full latent state has a unique answer, but whose solver-facing view is missing exactly one necessary atomic fact.<n>The solver must request the missing information in natural language under a strict budget and then integrate the returned fact into an exact final answer.<n> MIRA-Math contains 2,310 generated instances from 22 typed mathematical families spanning algebra, probability, linear systems, discrete structures, signal processing, Markov chains, circuits, computation, and numerical boundary-value problems.
  arXiv  Detail & Related papers  (2026-07-08T13:23:56Z)
• PassiveQA: A Three-Action Framework for Epistemically Calibrated Question Answering via Supervised Finetuning [0.0]
  We study decision-aware query resolution under incomplete information, where a model must determine whether to Answer, Ask for clarification, or Abstain.<n>We propose PassiveQA, a three-action framework that aligns model behaviour with information sufficiency through supervised finetuning.<n>Our approach integrates structured information-state representations, knowledge graph-grounded context, and a finetuned planner that explicitly models missing variables and decision reasoning.
  arXiv  Detail & Related papers  (2026-04-06T09:54:16Z)
• Knowing What's Missing: Assessing Information Sufficiency in Question Answering [3.8786514101828167]
  We propose a structured Identify-then-Verify framework for robust sufficiency modeling.<n>We evaluate our method against established baselines across diverse multi-hop and factual QA datasets.
  arXiv  Detail & Related papers  (2025-12-06T15:58:22Z)
• Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information [21.562453754113072]
  Large Reasoning Models (LRMs) have demonstrated remarkable problem-solving abilities in mathematics.<n>We propose a new dataset consisting of two types of incomplete problems with diverse contexts.<n>Based on the dataset, our systematical evaluation of LRMs reveals their inability in proactively asking for information.
  arXiv  Detail & Related papers  (2025-08-15T06:42:00Z)
• Teaching Language Models To Gather Information Proactively [53.85419549904644]
  Large language models (LLMs) are increasingly expected to function as collaborative partners.<n>In this work, we introduce a new task paradigm: proactive information gathering.<n>We design a scalable framework that generates partially specified, real-world tasks, masking key information.<n>Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information.
  arXiv  Detail & Related papers  (2025-07-28T23:50:09Z)
• Localizing and Mitigating Errors in Long-form Question Answering [79.63372684264921]
  Long-form question answering (LFQA) aims to provide thorough and in-depth answers to complex questions, enhancing comprehension.<n>This work introduces HaluQuestQA, the first hallucination dataset with localized error annotations for human-written and model-generated LFQA answers.
  arXiv  Detail & Related papers  (2024-07-16T17:23:16Z)
• CLARINET: Augmenting Language Models to Ask Clarification Questions for Retrieval [52.134133938779776]
  We present CLARINET, a system that asks informative clarification questions by choosing questions whose answers would maximize certainty in the correct candidate. Our approach works by augmenting a large language model (LLM) to condition on a retrieval distribution, finetuning end-to-end to generate the question that would have maximized the rank of the true candidate at each turn.
  arXiv  Detail & Related papers  (2024-04-28T18:21:31Z)
• UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models [55.22048505787125]
  This paper contributes a comprehensive dataset, called UNK-VQA. We first augment the existing data via deliberate perturbations on either the image or question. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models.
  arXiv  Detail & Related papers  (2023-10-17T02:38:09Z)
• CLAM: Selective Clarification for Ambiguous Questions with Large Language Models [37.37606905433334]
  We show that current SotA models do not ask the user for clarification when presented with imprecise questions. We introduce CLAM, a framework that first uses the model to detect ambiguous questions and if an ambiguous question is detected, prompts the model to ask the user for clarification. We show that our method achieves a 20.15 percentage point accuracy improvement over SotA on a novel ambiguous question-answering answering data set.
  arXiv  Detail & Related papers  (2022-12-15T12:47:18Z)
• Selective Question Answering under Domain Shift [90.021577320085]
  Abstention policies based solely on the model's softmax probabilities fare poorly, since models are overconfident on out-of-domain inputs. We train a calibrator to identify inputs on which the QA model errs, and abstain when it predicts an error is likely. Our method answers 56% of questions while maintaining 80% accuracy; in contrast, directly using the model's probabilities only answers 48% at 80% accuracy.
  arXiv  Detail & Related papers  (2020-06-16T19:13:21Z)
• ManyModalQA: Modality Disambiguation and QA over Diverse Inputs [73.93607719921945]
  We present a new multimodal question answering challenge, ManyModalQA, in which an agent must answer a question by considering three distinct modalities. We collect our data by scraping Wikipedia and then utilize crowdsourcing to collect question-answer pairs.
  arXiv  Detail & Related papers  (2020-01-22T14:39:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.