Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents

• URL: http://arxiv.org/abs/2603.26233v1
• Date: Fri, 27 Mar 2026 09:56:26 GMT
• Title: Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents
• Abstract summary: Large Language Model (LLM) agents are increasingly deployed in open-ended domains like software engineering.<n>We evaluate the clarification-seeking abilities of LLM agents on an underspecified variant of SWE-bench Verified.<n>We propose an uncertainty-aware multi-agent scaffold that explicitly decouples underspecification detection from code execution.
• Score: 4.301199871195023
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: As Large Language Model (LLM) agents are increasingly deployed in open-ended domains like software engineering, they frequently encounter underspecified instructions that lack crucial context. While human developers naturally resolve underspecification by asking clarifying questions, current agents are largely optimized for autonomous execution. In this work, we systematically evaluate the clarification-seeking abilities of LLM agents on an underspecified variant of SWE-bench Verified. We propose an uncertainty-aware multi-agent scaffold that explicitly decouples underspecification detection from code execution. Our results demonstrate that this multi-agent system using OpenHands + Claude Sonnet 4.5 achieves a 69.40% task resolve rate, significantly outperforming a standard single-agent setup (61.20%) and closing the performance gap with agents operating on fully specified instructions. Furthermore, we find that the multi-agent system exhibits well-calibrated uncertainty, conserving queries on simple tasks while proactively seeking information on more complex issues. These findings indicate that current models can be turned into proactive collaborators, where agents independently recognize when to ask questions to elicit missing information in real-world, underspecified tasks.

Related papers

• The Optimizer Is the Agent: Reasoning-Driven Search across Prompts, Programs, and ML Workflows [72.9051182568928]
  We present ReASearch, a unified framework for reasoning-driven optimization.<n>With a shared agent loop and domain-specific tools, ReASearch instantiates the exact same scaffold to optimize prompts, programs, and ML.<n>We observe that complex search behaviors emerge naturally from the agent's reasoning process.
  arXiv  Detail & Related papers  (2026-08-07T02:15:39Z)
• Formal Verification of Agentic Systems over Operational Data [59.98246281888422]
  Agentic systems driven by large language models (LLMs) are increasingly deployed in real-world where they act on persistent operational data.<n>Existing approaches do not provide such system-level guarantees.<n>We study here the verification of agentic systems comprising a single LLM and a tool orchestration harness over relational operational data.
  arXiv  Detail & Related papers  (2026-08-04T13:01:30Z)
• Beyond Generalist LLMs: Specialist Agentic Systems for Structured Code Workflow Execution [0.40631409309544836]
  Large Language Models (LLMs) have accelerated the adoption of software development agents, now widely available as Integrated Development Environment (IDE) extensions and standalone applications.<n>While these agents are typically general-purpose, it remains unclear whether specialist agents justify their additional development effort.<n>We investigate this question in the context of business process automation, focusing on the transformation of Business Process Model and Notation (BPMN) diagrams into executable agentic.
  arXiv  Detail & Related papers  (2026-07-16T01:06:38Z)
• AgentAbstain: Do LLM Agents Know When Not to Act? [19.223843408018997]
  Existing evaluations mostly focus on task success rather than whether agents know when to abstain.<n>We present the first systematic evaluation framework for agentic abstention: the calibrated ability of tool-using LLM agents to recognize when not to act.<n>AgentAbstain is a paired-task benchmark built on an agent-native taxonomy of 8 abstention scenarios across pre-execution reasoning and runtime discovery.
  arXiv  Detail & Related papers  (2026-07-11T00:57:36Z)
• AgentSearchBench: A Benchmark for AI Agent Search in the Wild [21.739444037726418]
  We introduce AgentSearchBench, a large-scale benchmark for agent search in the wild.<n>The benchmark formalizes agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions.<n>We show that lightweight behavioral signals, including execution-aware probing, can substantially improve ranking quality.
  arXiv  Detail & Related papers  (2026-04-24T10:53:54Z)
• AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios [49.90735676070039]
  The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow.<n>We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks.<n>We propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks.
  arXiv  Detail & Related papers  (2026-01-28T13:49:18Z)
• The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution [63.61358761489141]
  Large Language Model (LLM)-based agents are widely used in real-world applications such as customer service, web navigation, and software engineering.<n>We propose a novel framework for textbfgeneral agentic attribution, designed to identify the internal factors driving agent actions regardless of the task outcome.<n>We validate our framework across a diverse suite of agentic scenarios, including standard tool use and subtle reliability risks like memory-induced bias.
  arXiv  Detail & Related papers  (2026-01-21T15:22:21Z)
• AgentAsk: Multi-Agent Systems Need to Ask [26.13279490836716]
  Multi-agent systems built on large language models (LLMs) promise enhanced problem-solving capabilities through collaborative division of labor.<n>We propose AgentAsk, a lightweight and plug-and-play clarification module that treats every inter-agent message as a potential failure point and inserts minimally necessary questions to arrest error propagation.<n>AgentAsk consistently improves accuracy and robustness over public multi-agent implementations while keeping overhead minimal, with latency and extra cost all less than 5%.
  arXiv  Detail & Related papers  (2025-10-08T22:36:05Z)
• AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios [51.46347732659174]
  Large Language Models (LLMs) have demonstrated advanced capabilities in real-world agentic applications.<n>AgentIF is the first benchmark for systematically evaluating LLM instruction following ability in agentic scenarios.
  arXiv  Detail & Related papers  (2025-05-22T17:31:10Z)
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems [50.29939179830491]
  Failure attribution in LLM multi-agent systems remains underexplored and labor-intensive.<n>We develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons.<n>The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps.
  arXiv  Detail & Related papers  (2025-04-30T23:09:44Z)
• Defining and Detecting the Defects of the Large Language Model-based Autonomous Agents [31.126001253902416]
  We present the first study focused on identifying and detecting defects in LLM Agents.<n>We collected and analyzed 6,854 relevant posts from StackOverflow to define 8 types of agent defects.<n>Our results show that Agentable achieved an overall accuracy of 88.79% and a recall rate of 91.03%.
  arXiv  Detail & Related papers  (2024-12-24T11:54:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.