SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model

• URL: http://arxiv.org/abs/2606.14574v1
• Date: Fri, 12 Jun 2026 15:53:16 GMT
• Title: SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model
• Abstract summary: We introduce SIMMER, a benchmark for evaluating latent failures in large language models (LLMs) planning.<n> SIMMER defines a world model comprising 77 actions, 262 unique objects, and approximately 46,800 possible interactions that are semantically realistic.<n>It then leverages a state machine executor that validates plans against the world model and detects immediate precondition violations, latent hazards, and irreversible failures.
• Score: 8.782590561222511
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly deployed as planners for autonomous agents in household environments. While existing benchmarks evaluate whether LLM-generated plans execute successfully, they overlook a critical type of failure: latent failures. Unlike immediate failures that trigger instant feedback at execution time and enable timely correction, latent failures do not immediately halt plan execution but silently compromise goal achievement. In severe cases, they cause irreversible harm. To address this gap, we introduce SIMMER, a benchmark for evaluating latent failures in LLM planning through a human-curated symbolic world model grounded in the kitchen domain. SIMMER defines a world model comprising 77 actions, 262 unique objects, and approximately 46,800 possible interactions that are semantically realistic, derived from real-world cooking scripts. It then leverages a state machine executor that validates plans against the world model and detects immediate precondition violations, latent hazards, and irreversible failures. Experiments across six LLMs show that even frontier models achieve at most 17% error-free plans. Moreover, up to 56% of plans contain latent failures, the majority of which lead to irreversible consequences. We further demonstrate that explicit state reasoning via counterfactual foresight simulation can reduce latent failures by up to 72% and irreversible cases by up to 75%, suggesting a promising direction for more robust LLM planners.

Related papers

• AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints [65.27107190916512]
  We introduce AdaPlanBench, an interactive benchmark for evaluating whether Large Language Model (LLM) agents can adaptively plan and re-plan under progressively revealed constraints.<n>AdaPlanBench is built on 307 household tasks, with a scalable constraint construction pipeline that augments each task with dual constraints.<n>Experiments show that adaptive planning under dual constraints remains challenging, with the best model reaching only 67.75% accuracy.
  arXiv  Detail & Related papers  (2026-06-04T02:47:29Z)
• Characterizing the Failure Modes of LLMs in Resolving Real-World GitHub Issues [10.34693262597204]
  Large Language Models (LLMs) are increasingly deployed to resolve real-world GitHub issues.<n>This paper evaluates three state-of-the-art models, i.e., Claude 4.5 Sonnet, Gemini 3 Pro, and GPT-5, on the SWE-bench Verified dataset.<n>We conduct a rigorous manual analysis of the symptoms and root causes underlying 243 failed attempts across 900 total trials.
  arXiv  Detail & Related papers  (2026-05-12T15:34:57Z)
• DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policies [65.27025563507961]
  We propose DreamAvoid, a critical-phase test-time dreaming framework for Vision-Language-Action (VLA) models.<n>We also introduce an autonomous boundary learning paradigm to refine the system's understanding of the subtle boundary between success and failure.<n>The results demonstrate that DreamAvoid can effectively avoid failures, thereby improving the overall task success rate.
  arXiv  Detail & Related papers  (2026-05-12T08:27:16Z)
• PRISM: : Planning and Reasoning with Intent in Simulated Embodied Environments [59.07829883257003]
  Built on five multi-room apartments, PRISM structures 300 human-verified tasks into three capability tiers.<n>PRISM exposes an agent-agnostic executable action API that allows arbitrary agents to be evaluated end-to-end.
  arXiv  Detail & Related papers  (2026-05-12T04:59:47Z)
• Agentic LLM Planning via Step-Wise PDDL Simulation: An Empirical Characterisation [1.2832858109291982]
  Large language models (LLMs) can serve as viable planners alongside classical symbolic methods.<n>We present PyPDDLEngine, an open-source Planning Domain Definition Language (PDDL) simulation engine.<n>We evaluate four approaches on 102 International Planning Competition (IPC) Blocksworld instances under a uniform 180-second budget.
  arXiv  Detail & Related papers  (2026-03-06T09:16:49Z)
• VLM-DEWM: Dynamic External World Model for Verifiable and Resilient Vision-Language Planning in Manufacturing [6.758047936249273]
  Vision-language model (VLM) shows promise for high-level planning in smart manufacturing.<n>This paper presents VLM-DEWM, a cognitive architecture that decouples VLM reasoning from world-state management.<n>We evaluate VLM-DEWM on multi-station assembly, large-scale facility exploration, and real-robot recovery under induced failures.
  arXiv  Detail & Related papers  (2026-02-17T12:54:18Z)
• Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models [53.20969621498248]
  We propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures.<n>We construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail.<n>We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection.
  arXiv  Detail & Related papers  (2025-12-01T17:57:27Z)
• Scaling LLM Planning: NL2FLOW for Parametric Problem Generation and Rigorous Evaluation [0.913755431537592]
  This work introduces NL2Flow, a fully automated pipeline for generating and evaluating workflow planning problems.<n> NL2Flow generates problems parametrically in a structured intermediate representation, translating them into both natural language and formal PDDL.<n>I evaluate several open-source, instruct-tuned LLMs on a dataset of 2296 low-difficulty problems generated by NL2Flow.
  arXiv  Detail & Related papers  (2025-07-03T03:02:49Z)
• Self-Corrective Task Planning by Inverse Prompting with Large Language Models [9.283971287618261]
  We introduce InversePrompt, a novel self-corrective task planning approach.<n>Our method incorporates reasoning steps to provide clear, interpretable feedback.<n>Results on benchmark datasets show an average 16.3% higher success rate over existing LLM-based task planning methods.
  arXiv  Detail & Related papers  (2025-03-10T13:35:51Z)
• Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Planners [10.746821861109176]
  Large Language Models (LLMs) have witnessed remarkable performance as zero-shot task planners for robotic tasks.<n>However, the open-loop nature of previous works makes LLM-based planning error-prone and fragile.<n>In this work, we introduce a framework for closed-loop LLM-based planning called KnowLoop, backed by an uncertainty-based MLLMs failure detector.
  arXiv  Detail & Related papers  (2024-06-01T12:52:06Z)
• PDDLEGO: Iterative Planning in Textual Environments [56.12148805913657]
  Planning in textual environments has been shown to be a long-standing challenge even for current models. We propose PDDLEGO that iteratively construct a planning representation that can lead to a partial plan for a given sub-goal. We show that plans produced by few-shot PDDLEGO are 43% more efficient than generating plans end-to-end on the Coin Collector simulation.
  arXiv  Detail & Related papers  (2024-05-30T08:01:20Z)
• AdaPlanner: Adaptive Planning from Feedback with Language Models [56.367020818139665]
  Large language models (LLMs) have recently demonstrated the potential in acting as autonomous agents for sequential decision-making tasks. We propose a closed-loop approach, AdaPlanner, which allows the LLM agent to refine its self-generated plan adaptively in response to environmental feedback. To mitigate hallucination, we develop a code-style LLM prompt structure that facilitates plan generation across a variety of tasks, environments, and agent capabilities.
  arXiv  Detail & Related papers  (2023-05-26T05:52:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.