Bridging Learned Visual Perception and Symbolic Belief-Space Planning

• URL: http://arxiv.org/abs/2609.16884v1
• Date: Tue, 15 Sep 2026 09:17:32 GMT
• Title: Bridging Learned Visual Perception and Symbolic Belief-Space Planning
• Abstract summary: In partially observable settings, agents must act without full knowledge of the world state and rely on uncertain state-estimation pipelines.<n>Recent work has integrated Vision-Language Models (VLMs) to bridge perception and symbolic reasoning.<n>We introduce a novel approach that captures the uncertainty of VLM predicate groundings as a probability distribution over symbolic states.
• Score: 7.9081587430258145
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In partially observable settings, agents must act without full knowledge of the world state and rely on uncertain state-estimation pipelines. Obtaining grounded and verifiable symbolic plans under such uncertainty remains a key challenge. Recent work has integrated Vision-Language Models (VLMs) to bridge perception and symbolic reasoning, following two main paradigms. The first, VLM-as-planner, maps images directly to action sequences, and the second, VLM-as-grounder, grounds observations into symbolic predicates used as the initial state by off-the-shelf planners. Both approaches ignore uncertainty in the planning process, compromising robustness. We introduce a third paradigm, VLM-as-probabilistic-grounder, a novel approach that captures the uncertainty of VLM predicate groundings as a probability distribution over symbolic states. This enables planning in belief space and producing robust plans under uncertainty. Experiments in simulated household robot settings show improved robustness and task success over deterministic grounding, underscoring how our approach leverages foundation models for reliable planning under uncertainty.

Related papers

• Neurosymbolic Embodied Agents [47.49426471809857]
  Language and vision-language models generate plausible embodied plans but do not guarantee executability.<n>We present a neurosymbolic agent that factors long-horizon household tasks into task-directed visual exploration and constrained symbolic planning.
  arXiv  Detail & Related papers  (2026-08-17T16:50:59Z)
• Shift & Drift: A Zero-Shot Benchmark for Generalizable and Robust Autonomous Driving Motion Planning [60.888463449435484]
  We present Shift & Drift, a novel dual-track benchmark designed to rigorously stress-test motion planners across two critical axes of distribution shift.<n>We show that imitation learning methods achieve high scores in ID benchmarks, but suffer from persistent drift when subjected to temporally correlated actuation noise.<n>Our findings reveal an empirical trade-off between imitation fidelity and closed-loop resilience, providing the community with a rigorous benchmark to evaluate progress toward reliable deployment.
  arXiv  Detail & Related papers  (2026-07-08T18:23:48Z)
• ThinkingVLA: Interleaved Vision and Language Reasoning for Robotic Manipulation [71.69632267962993]
  We argue that manipulation planning naturally decomposes into prediction, anticipating the next visual state, and inverse dynamics.<n>We propose textbfThinkingVLA, a generative model that realizes this decomposition within a unified Mixture-of-Transformers architecture.<n>Experiments on simulation and real-world benchmarks demonstrate that ThinkingVLA consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-16T13:45:17Z)
• From General Vision to Reliable Traversability Estimation: Adapting Vision Foundation Models for Unstructured Outdoor Environments [12.275095764403432]
  We propose Vision-to-Traversability Adaptation (ViTA), a framework that adapts vision-based approaches for reliable traversability estimation.<n>ViTA injects task-specific knowledge through learnable traversability prompts while preserving the VFM's cross-domain generalization.<n>We show that ViTA achieves state-of-the-art IoU and Precision with substantial false-positive reduction and strong cross-domain generalization.
  arXiv  Detail & Related papers  (2026-05-28T08:16:20Z)
• PRISM: : Planning and Reasoning with Intent in Simulated Embodied Environments [59.07829883257003]
  Built on five multi-room apartments, PRISM structures 300 human-verified tasks into three capability tiers.<n>PRISM exposes an agent-agnostic executable action API that allows arbitrary agents to be evaluated end-to-end.
  arXiv  Detail & Related papers  (2026-05-12T04:59:47Z)
• Into the Unknown: Towards using Generative Models for Sampling Priors of Environment Uncertainty for Planning in Configuration Spaces [28.37021202108478]
  Priors are vital for planning under partial observability, yet difficult to obtain in practice.<n>We present a probabilistic-based pipeline that leverages large-scale pretrained generative models to produce priors in a zero-shot manner.<n>We establish a Matterport3D benchmark of rooms partially visible through doorways, where a robot must navigate to an unobserved target object.
  arXiv  Detail & Related papers  (2025-10-13T05:08:48Z)
• Towards Reliable LLM-based Robot Planning via Combined Uncertainty Estimation [68.106428321492]
  Large language models (LLMs) demonstrate advanced reasoning abilities, enabling robots to understand natural language instructions and generate high-level plans with appropriate grounding.<n>LLMs hallucinations present a significant challenge, often leading to overconfident yet potentially misaligned or unsafe plans.<n>We present Combined Uncertainty estimation for Reliable Embodied planning (CURE), which decomposes the uncertainty into epistemic and intrinsic uncertainty, each estimated separately.
  arXiv  Detail & Related papers  (2025-10-09T10:26:58Z)
• Seeing is Believing: Belief-Space Planning with Foundation Models as Uncertainty Estimators [34.28879194786174]
  Generalizable robotic mobile manipulation in open-world environments poses significant challenges due to long horizons, complex goals, and partial observability.<n>A promising approach to address these challenges involves planning with a library of parameterized skills, where a task planner sequences these skills to achieve goals specified in structured languages.<n>This paper introduces a novel framework that leverages vision-language models to estimate uncertainty and facilitate symbolic grounding.
  arXiv  Detail & Related papers  (2025-04-04T07:48:53Z)
• VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility [31.50489359729733]
  VISO-Grasp is a vision-informed system designed to address visibility constraints for grasping in severely occluded environments.<n>We introduce a multi-view uncertainty-driven grasp fusion mechanism that refines grasp confidence and directional uncertainty in real-time.<n>VISO-Grasp achieves a success rate of $87.5%$ in target-oriented grasping with the fewest grasp attempts outperforming baselines.
  arXiv  Detail & Related papers  (2025-03-16T18:46:54Z)
• Know Where You're Uncertain When Planning with Multimodal Foundation Models: A Formal Framework [54.40508478482667]
  We present a comprehensive framework to disentangle, quantify, and mitigate uncertainty in perception and plan generation.<n>We propose methods tailored to the unique properties of perception and decision-making.<n>We show that our uncertainty disentanglement framework reduces variability by up to 40% and enhances task success rates by 5% compared to baselines.
  arXiv  Detail & Related papers  (2024-11-03T17:32:00Z)
• Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguity [0.659529078336196]
  Large language models (LLMs) exhibit advanced reasoning skills, enabling robots to comprehend natural language instructions and strategically plan high-level actions.<n>LLMs hallucination may result in robots confidently executing plans that are misaligned with user goals or even unsafe in critical scenarios.<n>We propose introspective planning, a systematic approach that align LLM's uncertainty with the inherent ambiguity of the task.
  arXiv  Detail & Related papers  (2024-02-09T16:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.