Clarification Is Not Enough: Post-Clarification Answering Remains the Bottleneck in Multi-Turn QA

• URL: http://arxiv.org/abs/2605.25204v1
• Date: Sun, 24 May 2026 18:06:09 GMT
• Title: Clarification Is Not Enough: Post-Clarification Answering Remains the Bottleneck in Multi-Turn QA
• Abstract summary: We study the problem of preference elicitation in multi-turn question answering by decomposing the problem into two components.<n>We show, using the PACIFIC benchmark, that supervised fine-tuning rapidly improves the clarification policy, however, final answer accuracy remains substantially lower even when the model takes the correct action.
• Score: 14.475086992410276
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pluralistic alignment requires systems to adapt to diverse user values, communication styles, and contextual assumptions. We believe that a foundational prerequisite for such alignment enabling accurate preference elicitation from people when their intent is under-specified or ambiguous. We study the problem of preference elicitation in multi-turn question answering by decomposing the problem into two components: a \textbf{clarification policy}, which decides whether to ask a clarifying question or answer directly, and \textbf{post-clarification answering}, which produces the correct final answer once the missing information is provided. We show, using the PACIFIC benchmark, that supervised fine-tuning rapidly improves the clarification policy, however, final answer accuracy remains substantially lower even when the model takes the correct action. This gap indicates that understanding and correctly interpreting the user's response is the critical gap in multi-turn question-answering systems.

Related papers

• CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement [50.91045324738942]
  CLAIM is an uncertainty-driven framework for active clarification learning in open-domain settings.<n>We propose an entropy-driven synthetic data generation pipeline that integrates uncertainty estimation with semantic clustering and reasoning-based judgments.<n> Experimental results demonstrate that CLAIM can learn stable and generalizable clarification strategies without relying on manually labeled data.
  arXiv  Detail & Related papers  (2026-08-12T04:27:45Z)
• One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies [49.067670811577045]
  We introduce RegretBench, a benchmark that evaluates clarification as policy behavior rather than isolated question quality.<n>RegretBench measures intent resolution, interaction cost, ineffective clarification, and regret.<n>Our results show that effective clarification requires more than plausible questions.
  arXiv  Detail & Related papers  (2026-07-23T10:22:11Z)
• Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions [17.694274559676867]
  We evaluate models on ambiguous, unambiguous, and disambiguated questions in three settings.<n>We find a clear gap between recognition and behavior: models often identify ambiguity when explicitly asked to judge it, yet in the QA setting they overwhelmingly default to direct answers.
  arXiv  Detail & Related papers  (2026-05-24T22:36:58Z)
• When and What to Ask: AskBench and Rubric-Guided RLVR for LLM Clarification [8.391356566325054]
  Large language models (LLMs) often respond even when prompts omit critical details or include misleading information.<n>We study how to evaluate and improve LLMs' ability to decide when and what to ask for clarification without sacrificing task performance.<n>We introduce AskBench, an interactive benchmark that converts standard QA pairs into multi-turn interactions with explicit checkpoints.
  arXiv  Detail & Related papers  (2026-02-04T02:21:01Z)
• Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification [10.412378789389544]
  We propose CoA, an ask-or-answer agent that separately models the decision to ask or answer, and what to ask if needed.<n>CoA first determines whether clarification is necessary; if so, it asks a single focused question and then incorporates the response to produce the final answer.<n>CoA achieves consistent improvements at both the module and system levels, improving end-to-end VQA accuracy by an average of +15.3 points (83%) over prompting-based baselines.
  arXiv  Detail & Related papers  (2026-01-23T02:12:33Z)
• Learning Steerable Clarification Policies with Collaborative Self-play [67.67872810596839]
  To handle ambiguous queries, AI assistants need a policy for managing their uncertainty.<n>We propose to train steerable policies for managing this uncertainty using self-play.<n>We show this leads to a steerable policy that changes its behavior predictably conditioned on the provided costs.
  arXiv  Detail & Related papers  (2025-12-03T18:49:54Z)
• Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions [45.04582353648683]
  Large language models (LLMs) must often respond to highly ambiguous user requests.<n>Existing LLMs often respond by presupposing a single interpretation of such ambiguous requests, frustrating users who intended a different interpretation.<n>We propose preference labels by simulating their expected outcomes in future turns.<n>This allows LLMs to learn to ask clarifying questions when it can generate responses that are tailored to each user interpretation in future turns.
  arXiv  Detail & Related papers  (2024-10-17T17:29:04Z)
• CLARINET: Augmenting Language Models to Ask Clarification Questions for Retrieval [52.134133938779776]
  We present CLARINET, a system that asks informative clarification questions by choosing questions whose answers would maximize certainty in the correct candidate. Our approach works by augmenting a large language model (LLM) to condition on a retrieval distribution, finetuning end-to-end to generate the question that would have maximized the rank of the true candidate at each turn.
  arXiv  Detail & Related papers  (2024-04-28T18:21:31Z)
• Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs [58.620269228776294]
  We propose a task-agnostic framework for resolving ambiguity by asking users clarifying questions. We evaluate systems across three NLP applications: question answering, machine translation and natural language inference. We find that intent-sim is robust, demonstrating improvements across a wide range of NLP tasks and LMs.
  arXiv  Detail & Related papers  (2023-11-16T00:18:50Z)
• From Chaos to Clarity: Claim Normalization to Empower Fact-Checking [57.024192702939736]
  Claim Normalization (aka ClaimNorm) aims to decompose complex and noisy social media posts into more straightforward and understandable forms. We propose CACN, a pioneering approach that leverages chain-of-thought and claim check-worthiness estimation. Our experiments demonstrate that CACN outperforms several baselines across various evaluation measures.
  arXiv  Detail & Related papers  (2023-10-22T16:07:06Z)
• Answering Ambiguous Questions via Iterative Prompting [84.3426020642704]
  In open-domain question answering, due to the ambiguity of questions, multiple plausible answers may exist. One approach is to directly predict all valid answers, but this can struggle with balancing relevance and diversity. We present AmbigPrompt to address the imperfections of existing approaches to answering ambiguous questions.
  arXiv  Detail & Related papers  (2023-07-08T04:32:17Z)
• CLAM: Selective Clarification for Ambiguous Questions with Large Language Models [37.37606905433334]
  We show that current SotA models do not ask the user for clarification when presented with imprecise questions. We introduce CLAM, a framework that first uses the model to detect ambiguous questions and if an ambiguous question is detected, prompts the model to ask the user for clarification. We show that our method achieves a 20.15 percentage point accuracy improvement over SotA on a novel ambiguous question-answering answering data set.
  arXiv  Detail & Related papers  (2022-12-15T12:47:18Z)
• Double Retrieval and Ranking for Accurate Question Answering [120.69820139008138]
  We show that an answer verification step introduced in Transformer-based answer selection models can significantly improve the state of the art in Question Answering. The results on three well-known datasets for AS2 show consistent and significant improvement of the state of the art.
  arXiv  Detail & Related papers  (2022-01-16T06:20:07Z)
• Answering Ambiguous Questions through Generative Evidence Fusion and Round-Trip Prediction [46.38201136570501]
  We present a model that aggregates and combines evidence from multiple passages to adaptively predict a single answer or a set of question-answer pairs for ambiguous questions. Our model, named Refuel, achieves a new state-of-the-art performance on the AmbigQA dataset, and shows competitive performance on NQ-Open and TriviaQA.
  arXiv  Detail & Related papers  (2020-11-26T05:48:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.