Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions

• URL: http://arxiv.org/abs/2605.25284v1
• Date: Sun, 24 May 2026 22:36:58 GMT
• Title: Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
• Abstract summary: We evaluate models on ambiguous, unambiguous, and disambiguated questions in three settings.<n>We find a clear gap between recognition and behavior: models often identify ambiguity when explicitly asked to judge it, yet in the QA setting they overwhelmingly default to direct answers.
• Score: 17.694274559676867
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: User queries are often underspecified and may admit multiple valid interpretations. Rather than silently making assumptions about the user's intent, a helpful assistant should surface such ambiguity by asking a clarifying question. Doing so requires two abilities: recognizing that a query is ambiguous, and acting on that recognition by seeking clarification instead of answering directly. To study these abilities, we evaluate models on ambiguous, unambiguous, and disambiguated questions in three settings: standard question answering, explicit ambiguity judgment, and behavioral analysis, where a judge model classifies responses as direct answers, refusals, or clarifying questions. We find a clear gap between recognition and behavior: models often identify ambiguity when explicitly asked to judge it, yet in the QA setting they overwhelmingly default to direct answers. Retrieved context further widens this gap by improving answerability while making models even less likely to ask clarifying questions.

Related papers

• FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect [69.8983512616053]
  Large language models often change their responses to subtle rephrasings that align with an implied stance by users.<n>This can leave users with advice tainted by how they happened to phrase a question rather than by the underlying facts.<n>We introduce FramingQA, a benchmark that measures the model sensitivity to question framing across law, medicine, finance, and robotic simulations.
  arXiv  Detail & Related papers  (2026-09-07T12:56:44Z)
• One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies [49.067670811577045]
  We introduce RegretBench, a benchmark that evaluates clarification as policy behavior rather than isolated question quality.<n>RegretBench measures intent resolution, interaction cost, ineffective clarification, and regret.<n>Our results show that effective clarification requires more than plausible questions.
  arXiv  Detail & Related papers  (2026-07-23T10:22:11Z)
• Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions [24.349064784305316]
  In visual question answering (VQA) context, users often pose ambiguous questions to visual language models (VLMs) due to varying expression habits.<n>We introduce bftextClearVQA benchmark, which targets three common categories of ambiguity in VQA context.
  arXiv  Detail & Related papers  (2025-07-18T09:31:43Z)
• Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations [70.6395572287422]
  Self-alignment method is capable of not only refusing to answer but also providing explanation to the unanswerability of unknown questions. We conduct disparity-driven self-curation to select qualified data for fine-tuning the LLM itself for aligning the responses to unknown questions as desired.
  arXiv  Detail & Related papers  (2024-02-23T02:24:36Z)
• Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs [58.620269228776294]
  We propose a task-agnostic framework for resolving ambiguity by asking users clarifying questions. We evaluate systems across three NLP applications: question answering, machine translation and natural language inference. We find that intent-sim is robust, demonstrating improvements across a wide range of NLP tasks and LMs.
  arXiv  Detail & Related papers  (2023-11-16T00:18:50Z)
• Open-ended Commonsense Reasoning with Unrestricted Answer Scope [47.14397700770702]
  Open-ended Commonsense Reasoning is defined as solving a commonsense question without providing 1) a short list of answer candidates and 2) a pre-defined answer scope. In this work, we leverage pre-trained language models to iteratively retrieve reasoning paths on the external knowledge base. The reasoning paths can help to identify the most precise answer to the commonsense question.
  arXiv  Detail & Related papers  (2023-10-18T02:45:54Z)
• Answering Ambiguous Questions via Iterative Prompting [84.3426020642704]
  In open-domain question answering, due to the ambiguity of questions, multiple plausible answers may exist. One approach is to directly predict all valid answers, but this can struggle with balancing relevance and diversity. We present AmbigPrompt to address the imperfections of existing approaches to answering ambiguous questions.
  arXiv  Detail & Related papers  (2023-07-08T04:32:17Z)
• Selectively Answering Ambiguous Questions [38.83930394700588]
  We find that the most reliable approach to decide when to abstain involves quantifying repetition within sampled model outputs. Our results suggest that sampling-based confidence scores help calibrate answers to relatively unambiguous questions.
  arXiv  Detail & Related papers  (2023-05-24T01:25:38Z)
• CLAM: Selective Clarification for Ambiguous Questions with Large Language Models [37.37606905433334]
  We show that current SotA models do not ask the user for clarification when presented with imprecise questions. We introduce CLAM, a framework that first uses the model to detect ambiguous questions and if an ambiguous question is detected, prompts the model to ask the user for clarification. We show that our method achieves a 20.15 percentage point accuracy improvement over SotA on a novel ambiguous question-answering answering data set.
  arXiv  Detail & Related papers  (2022-12-15T12:47:18Z)
• Deciding Whether to Ask Clarifying Questions in Large-Scale Spoken Language Understanding [28.195853603190447]
  A large-scale conversational agent can suffer from understanding user utterances with various ambiguities. We propose a neural self-attentive model that leverages the hypotheses with ambiguities and contextual signals.
  arXiv  Detail & Related papers  (2021-09-25T22:32:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.