AsgardBench -- Evaluating Visually Grounded Interactive Planning Under Minimal Feedback

• URL: http://arxiv.org/abs/2603.15888v2
• Date: Wed, 18 Mar 2026 14:41:47 GMT
• Title: AsgardBench -- Evaluating Visually Grounded Interactive Planning Under Minimal Feedback
• Abstract summary: AsgardBench aims to evaluate visually grounded, high-level action sequence generation and interactive planning.<n>Our evaluations show that performance drops sharply without visual input, revealing weaknesses in visual grounding and state tracking.<n>Our benchmark zeroes in on a narrower question: can a model actually use what it sees to adapt a plan when things do not go as expected?
• Score: 28.254110943200683
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: With AsgardBench we aim to evaluate visually grounded, high-level action sequence generation and interactive planning, focusing specifically on plan adaptation during execution based on visual observations rather than navigation or low-level manipulation. In the landscape of embodied AI benchmarks, AsgardBench targets the capability category of interactive planning, which is more sophisticated than offline high-level planning as it requires agents to revise plans in response to environmental feedback, yet remains distinct from low-level execution. Unlike prior embodied AI benchmarks that conflate reasoning with navigation or provide rich corrective feedback that substitutes for perception, AsgardBench restricts agent input to images, action history, and lightweight success/failure signals, isolating interactive planning in a controlled simulator without low-level control noise. The benchmark contains 108 task instances spanning 12 task types, each systematically varied through object state, placement, and scene configuration. These controlled variations create conditional branches in which a single instruction can require different action sequences depending on what the agent observes, emphasizing conditional branching and plan repair during execution. Our evaluations of leading vision language models show that performance drops sharply without visual input, revealing weaknesses in visual grounding and state tracking that ultimately undermine interactive planning. Our benchmark zeroes in on a narrower question: can a model actually use what it sees to adapt a plan when things do not go as expected?

Related papers

• Action- and Language-Conditioned Video Assessment for Embodied Control [25.788621525752006]
  Vision-based embodied agents executing natural language instructions require feedback mechanisms that assess task progress over complete trajectories.<n>We propose ALVA (Action- and Language-Conditioned Video Assessment), a trajectory evaluator that conditions its assessment on visual observations, the executed action sequence, and the natural language instruction.<n>In simulated 3D household environments, ALVA exhibits a conservative assessment pattern with near-zero false-positive rates.
  arXiv  Detail & Related papers  (2026-08-08T18:05:21Z)
• WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory [50.19449886428936]
  We introduce WorldScape Policy 2.0, a controllable WAM with reasoning-augmented long short-term memory.<n>The retrieved history augments perception and autoregressively generated planning tokens, yielding an implicit subgoal condition for autonomous planning.<n>Experiments in both simulation and real-world platforms demonstrate superior capabilities in long-horizon autonomous planning, fine-grained instruction following, and in-context adaptation.
  arXiv  Detail & Related papers  (2026-07-21T08:25:37Z)
• Foresight: Iterative Reasoning About Clues that Matter for Navigation [20.227441986534767]
  Mapless navigation requires resolving underspecified goals and inferring which environmental cues are relevant for reaching the goal.<n>Prior works are limited by their reliance on known navigation factors and closed-set factor categories, or identify cues before motion planning and miss plan-dependent cues.<n>We argue that pretrained Vision-Language Models (VLMs) can discover novel instruction-relevant cues, but require adaptation to focus on which cues matter and how they should influence motion planning.<n>We realize these ideas in Foresight, a test-time framework in which a finetuned VLM alternates between proposing image-space motion plans and critiquing
  arXiv  Detail & Related papers  (2026-06-10T18:01:06Z)
• Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models [101.44987067974999]
  Coarse-to-Control is a plan-execute VLA that introduces planning in the action-token space.<n>We show that action-token planning consistently improves over direct action generation.
  arXiv  Detail & Related papers  (2026-06-05T10:01:37Z)
• Afford-VLA: Action-Aligned Visual Planning via Internalized Affordance [108.46436073194546]
  We argue that effective planning should be local, visually grounded, internally generated, and directly aligned with action.<n>We propose Afford-VLA, a unified framework that internalizes task-conditioned affordance as an explicit visual planning interface within VLA models.
  arXiv  Detail & Related papers  (2026-05-22T20:43:47Z)
• From Perception to Action: An Interactive Benchmark for Vision Reasoning [51.11355591375073]
  Causal Hierarchy of Actions and Interactions (CHAIN) benchmark designed to evaluate whether models can understand, plan, and execute structured action sequences grounded in physical constraints.<n> CHAIN shifts evaluation from passive perception to active problem solving, spanning tasks such as interlocking mechanical puzzles and 3D stacking and packing.<n>Our results show that top-performing models still struggle to internalize physical structure and causal constraints, often failing to produce reliable long-horizon plans and cannot robustly translate perceived structure into effective actions.
  arXiv  Detail & Related papers  (2026-02-24T15:33:02Z)
• \textsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation [50.027425808733994]
  textscNaVIDA is a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution.<n>textscNaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions.<n>Experiments show that textscNaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters.
  arXiv  Detail & Related papers  (2026-01-26T06:16:17Z)
• Action-Sketcher: From Reasoning to Action via Visual Sketches for Long-Horizon Robotic Manipulation [37.76888706583038]
  Long-horizon robotic manipulation is increasingly important for real-world deployment.<n>Existing end-to-end and hierarchical Vision-Language-Action policies often rely on text-only cues.<n>We introduce Visual Sketch, an implausible visual intermediate that renders points, boxes, arrows, and typed relations in the robot's current views.<n>We present Action-Sketcher, a VLA framework that operates in a cyclic See-Think-Sketch-Act workflow.
  arXiv  Detail & Related papers  (2026-01-04T17:53:42Z)
• CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates [35.64639873984766]
  We propose a training-free method, Scene Graph Incremental updates (SGI), which introduces intermediate reasoning steps between the initial and goal states.<n>In addition to enhancing reliability in sequential planning, SGI generalizes to traditional planning tasks such as Plan-Bench and VQA.
  arXiv  Detail & Related papers  (2025-12-11T06:46:51Z)
• ATLAS: Actor-Critic Task-Completion with Look-ahead Action Simulation [28.54052846801967]
  ATLAS is a memory-augmented agent that makes plans grounded in a model of the environment by simulating the consequences of those actions in cognitive space.<n>On the WebArena-Lite Benchmark, we achieve a 63% success rate compared to 53.9% success rate for the previously published state-of-the-art.
  arXiv  Detail & Related papers  (2025-10-26T16:03:39Z)
• The Cognitive Bandwidth Bottleneck: Shifting Long-Horizon Agent from Planning with Actions to Planning with Schemas [56.62286434195321]
  This paper systematically studies the effectiveness of two different action representations.<n>We propose cognitive bandwidth perspective as a conceptual framework to qualitatively understand the differences.<n>We provide an actionable guide for building more capable PwS agents for better scalable autonomy.
  arXiv  Detail & Related papers  (2025-10-08T14:47:40Z)
• Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agent [9.370683025542686]
  We present a proactive replanning framework that detects and corrects failures at subtask boundaries.<n>Experiments in the AI2-THOR simulator demonstrate that our approach detects semantic and spatial mismatches before execution failures occur.
  arXiv  Detail & Related papers  (2025-08-15T07:48:51Z)
• ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning [47.27336786187929]
  Vision-language-action (VLA) reasoning tasks require agents to interpret multimodal instructions, perform long-horizon planning, and act adaptively in dynamic environments.<n>Existing approaches typically train VLA models in an end-to-end fashion, directly mapping inputs to actions without explicit reasoning.<n>We propose ThinkAct, a dual-system framework that bridges high-level reasoning with low-level action execution via reinforced visual latent planning.
  arXiv  Detail & Related papers  (2025-07-22T17:59:46Z)
• GTA1: GUI Test-time Scaling Agent [97.58177633084915]
  Graphical user interface (GUI) agents autonomously complete tasks across platforms (eg, Linux) by sequentially decomposing user instructions into action proposals.<n>This paper investigates the aforementioned challenges with our textbfGUI textbfTest-time Scaling textbfAgent, namely GTA1.
  arXiv  Detail & Related papers  (2025-07-08T08:52:18Z)
• Hallucinative Topological Memory for Zero-Shot Visual Planning [86.20780756832502]
  In visual planning (VP), an agent learns to plan goal-directed behavior from observations of a dynamical system obtained offline. Most previous works on VP approached the problem by planning in a learned latent space, resulting in low-quality visual plans. Here, we propose a simple VP method that plans directly in image space and displays competitive performance.
  arXiv  Detail & Related papers  (2020-02-27T18:54:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.