LHAW: Controllable Underspecification for Long-Horizon Tasks

• URL: http://arxiv.org/abs/2602.10525v1
• Date: Wed, 11 Feb 2026 04:49:50 GMT
• Title: LHAW: Controllable Underspecification for Long-Horizon Tasks
• Authors: George Pu, Michael S. Lee, Udari Madhushani Sehwag, David J. Lee, Bryan Zhu, Yash Maurya, Mohit Raghavendra, Yuan Xue, Samuel Marc Denton,
• Abstract summary: We introduce LHAW (Long-Horizon Augmenteds), a modular, dataset-agnostic synthetic pipeline that transforms any well-specified task into controllable underspecified variants.<n>Unlike approaches that rely on LLM predictions of ambiguity, LHAW validates variants through empirical agent trials, classifying them as outcome-critical, divergent, or benign based on observed terminal state divergence.<n>We release 285 task variants from TheAgentCompany, SWE-Bench Pro and MCP-Atlas, measuring how current agents detect, reason about, and resolve underspecification across ambiguous settings.
• Score: 8.46227536869596
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-horizon workflow agents that operate effectively over extended periods are essential for truly autonomous systems. Their reliable execution critically depends on the ability to reason through ambiguous situations in which clarification seeking is necessary to ensure correct task execution. However, progress is limited by the lack of scalable, task-agnostic frameworks for systematically curating and measuring the impact of ambiguity across custom workflows. We address this gap by introducing LHAW (Long-Horizon Augmented Workflows), a modular, dataset-agnostic synthetic pipeline that transforms any well-specified task into controllable underspecified variants by systematically removing information across four dimensions - Goals, Constraints, Inputs, and Context - at configurable severity levels. Unlike approaches that rely on LLM predictions of ambiguity, LHAW validates variants through empirical agent trials, classifying them as outcome-critical, divergent, or benign based on observed terminal state divergence. We release 285 task variants from TheAgentCompany, SWE-Bench Pro and MCP-Atlas according to our taxonomy alongside formal analysis measuring how current agents detect, reason about, and resolve underspecification across ambiguous settings. LHAW provides the first systematic framework for cost-sensitive evaluation of agent clarification behavior in long-horizon settings, enabling development of reliable autonomous systems.

Related papers

• Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems [0.0]
  We present a case-aware LLM-as-a-Judge evaluation framework for enterprise multi-turn RAG systems.<n>The framework evaluates each turn using eight operationally grounded metrics that separate retrieval quality, grounding fidelity, answer utility, precision integrity, and case/workflow alignment.
  arXiv  Detail & Related papers  (2026-02-23T21:37:06Z)
• Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance [92.72779885657373]
  We propose a framework that grounds model selection in the internal functional dynamics of the visual encoder.<n>Our approach represents each task via layer wise conductance and derives a target-conditioned block importance distribution through entropy regularized alignment.<n>Building on this, we introduce Directional Conductance Divergence (DCD), an asymmetric metric that quantifies how effectively a source task covers the target's salient functional blocks.
  arXiv  Detail & Related papers  (2026-02-01T17:29:43Z)
• LLM-Assisted Logic Rule Learning: Scaling Human Expertise for Time Series Anomaly Detection [0.9740025522928777]
  Time series anomaly detection is critical for supply chain management to take proactive operations.<n>We propose a framework that leverages large language models (LLMs) to systematically encode human expertise into interpretable, logic-based rules.
  arXiv  Detail & Related papers  (2026-01-27T06:37:37Z)
• LLM Performance Predictors: Learning When to Escalate in Hybrid Human-AI Moderation Systems [5.7001352660257005]
  We propose a framework for supervised uncertainty quantification in content moderation systems.<n>We show that our method enables cost-aware selective classification in real-world human-AI.<n>This work establishes a principled framework for uncertainty-aware, scalable and responsible human-AI moderation.
  arXiv  Detail & Related papers  (2026-01-11T17:46:49Z)
• Hypothesize-Then-Verify: Speculative Root Cause Analysis for Microservices with Pathwise Parallelism [19.31110304702373]
  SpecRCA is a speculative root cause analysis framework that adopts a textithypothesize-then-verify paradigm.<n>Preliminary experiments on the AIOps 2022 demonstrate that SpecRCA achieves superior accuracy and efficiency compared to existing approaches.
  arXiv  Detail & Related papers  (2026-01-06T05:58:25Z)
• Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking [64.97768177044355]
  Large Language Models (LLMs) are increasingly deployed in real-world fact-checking systems.<n>We present FactArena, a fully automated arena-style evaluation framework.<n>Our analyses reveal significant discrepancies between static claim-verification accuracy and end-to-end fact-checking competence.
  arXiv  Detail & Related papers  (2026-01-06T02:51:56Z)
• Reasoning with Confidence: Efficient Verification of LLM Reasoning Steps via Uncertainty Heads [104.9566359759396]
  We propose a lightweight alternative for step-level reasoning verification based on data-driven uncertainty scores.<n>Our findings suggest that the internal states of LLMs encode their uncertainty and can serve as reliable signals for reasoning verification.
  arXiv  Detail & Related papers  (2025-11-09T03:38:29Z)
• The Illusion of Procedural Reasoning: Measuring Long-Horizon FSM Execution in LLMs [10.228723521208858]
  Large language models (LLMs) have achieved remarkable results on tasks framed as reasoning problems.<n>Their true ability to perform procedural reasoning, executing multi-step, rule-based computations remains unclear.<n>We introduce Finite-State Machine Execution as a framework for evaluating the procedural reasoning capacity of LLMs.
  arXiv  Detail & Related papers  (2025-11-05T18:44:47Z)
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning [82.925106913459]
  Reinforcement finetuning (RFT) is a key technique for aligning Large Language Models (LLMs) with human preferences and enhancing reasoning.<n>We introduce BOTS, a unified framework for Bayesian Online Task Selection in RFT reinforcement finetuning.
  arXiv  Detail & Related papers  (2025-10-30T11:15:23Z)
• A Fano-Style Accuracy Upper Bound for LLM Single-Pass Reasoning in Multi-Hop QA [65.38186593873313]
  Multi-Hop Question Answering (MHQA) requires integrating dispersed, interdependent evidence through sequential reasoning under noise.<n>We introduce a proof-of-concept multi-call framework for MHQA, InfoQA.<n>We construct a stringent and noise-rich benchmark to validate our theory and framework.
  arXiv  Detail & Related papers  (2025-09-25T14:11:57Z)
• AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents [0.0]
  Large Language Model (LLM) agents become more widespread, associated misalignment risks increase.<n>In this work, we approach misalignment as a conflict between the internal goals pursued by the model and the goals intended by its deployer.<n>We introduce a misalignment propensity benchmark, textscAgentMisalignment, a benchmark suite designed to evaluate the propensity of LLM agents to misalign in realistic scenarios.
  arXiv  Detail & Related papers  (2025-06-04T14:46:47Z)
• Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making [85.24399869971236]
  We aim to evaluate Large Language Models (LLMs) for embodied decision making.<n>Existing evaluations tend to rely solely on a final success rate.<n>We propose a generalized interface (Embodied Agent Interface) that supports the formalization of various types of tasks.
  arXiv  Detail & Related papers  (2024-10-09T17:59:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.