Bridging Semantics and Physics with Constrained LLMs for Safe and Trustworthy Robotic Manipulation

• URL: http://arxiv.org/abs/2608.29379v1
• Date: Sat, 29 Aug 2026 17:26:20 GMT
• Title: Bridging Semantics and Physics with Constrained LLMs for Safe and Trustworthy Robotic Manipulation
• Abstract summary: A language-guided robot must do more than produce a plan that appears correct.<n>Plan may appear valid linguistically while being physically infeasible under kinematic and collision constraints.<n>We bridge this gap by formalizing the reasoning-execution boundary as a typed contract.
• Score: 6.975054201075641
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A language-guided robot operating in a real kitchen must do more than produce a plan that appears correct. It must also execute that plan safely in cluttered environments under imperfect perception. Large language models (LLM) can decompose instructions into action sequences, yet a language-action gap remains: a plan may appear valid linguistically while being physically infeasible under kinematic and collision constraints. We bridge this gap by formalizing the reasoning-execution boundary as a typed contract. From RGB-D observations, the system grounds perceived objects in an explicit, collision-aware scene model and constrains language-level decisions through schema-validated tool calls defined by the Model Context Protocol (MCP), rejecting malformed commands before they reach the robot. Each validated call is deterministically grounded in a MoveIt Task Constructor pipeline, where candidate motions are evaluated against the reconstructed planning scene in a verify-then-act step. Only trajectories that pass both kinematic and collision checks are sent to the robot. On a physical UFactory 850, the method achieves up to 80% success across ten trials per task on pouring tasks involving liquids, granular media, and discrete solids. It achieves 90% success on a grasp-and-place task using the same planning, protocol, and verification stack. Although a scripted policy slightly outperforms our method on the easiest task, its success rate falls to 10% on the hardest, compared with 60% for our method.

Related papers

• A Brain-inspired Hierarchical Framework for Zero-Shot Robot Task Reasoning and Execution [9.745144598922584]
  Method is a zero-shot hierarchical framework inspired by the division of roles in the human brain.<n>It grounds commands in explicit object states, composes reusable atomic actions into task-conditioned sequences, and ranks alternative sequences by execution cost.<n>Method achieves higher success rates than ReKep, Dream2Flow, and $_0.5$ benchmarks.
  arXiv  Detail & Related papers  (2026-09-05T08:59:04Z)
• ACE: Agentic Control for Embodied Manipulation via Zero-shot Workflow Reasoning [32.18940485472935]
  We present ACE, a zero-shot workflow reasoning framework for tabletop pick-and-place from natural language.<n>We evaluate ACE on logically complex, long-horizon tasks, including zero-shot multi-step equation formation with number cubes and constraint-based object retrieval.
  arXiv  Detail & Related papers  (2026-07-05T08:07:42Z)
• Do What? Teaching Vision-Language-Action Models to Reject the Impossible [53.40183895299108]
  Vision-Language-Action (VLA) models have demonstrated strong performance on a range of robotic tasks.<n>We propose Instruct-Verify-and-Act (IVA), a framework that detects when an instruction cannot be executed due to a false premise.<n>Our experiments show that IVA improves false premise detection accuracy by 97.56% over baselines.
  arXiv  Detail & Related papers  (2025-08-22T10:54:33Z)
• Context Matters! Relaxing Goals with LLMs for Feasible 3D Scene Planning [1.5391321019692432]
  Embodied agents need to plan and act reliably in real and complex 3D environments.<n>We introduce ContextMatters, a framework that fuses Large Language Models (LLMs) and classical planning.<n>Our experimental results show a +52.45% Success Rate improvement over state-of-the-art LLMs+PDDL baseline.
  arXiv  Detail & Related papers  (2025-06-18T19:14:56Z)
• Grounding Language Plans in Demonstrations Through Counterfactual Perturbations [25.19071357445557]
  Grounding the common-sense reasoning of Large Language Models (LLMs) in physical domains remains a pivotal yet unsolved problem for embodied AI. We show our approach improves the interpretability and reactivity of imitation learning through 2D navigation and simulated and real robot manipulation tasks.
  arXiv  Detail & Related papers  (2024-03-25T19:04:59Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• Language-Conditioned Path Planning [68.13248140217222]
  Language-Conditioned Collision Functions (LACO) learns a collision function using only a single-view image, language prompt, and robot configuration. LACO predicts collisions between the robot and the environment, enabling flexible, conditional path planning without the need for object annotations, point cloud data, or ground-truth object meshes. In both simulation and the real world, we demonstrate that LACO can facilitate complex, nuanced path plans that allow for interaction with objects that are safe to collide, rather than prohibiting any collision.
  arXiv  Detail & Related papers  (2023-08-31T17:56:13Z)
• DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment [10.322491116030825]
  DoReMi enables Detection and Recovery from Misalignments between plan and execution. Our pipeline can monitor the low-level execution and enable timely recovery if certain plan-execution misalignment occurs.
  arXiv  Detail & Related papers  (2023-07-01T12:51:02Z)
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models [68.57918965060787]
  Large language models (LLMs) can be used to score potential next actions during task planning. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments.
  arXiv  Detail & Related papers  (2022-09-22T20:29:49Z)
• Correcting Robot Plans with Natural Language Feedback [88.92824527743105]
  We explore natural language as an expressive and flexible tool for robot correction. We show that these transformations enable users to correct goals, update robot motions, and recover from planning errors. Our method makes it possible to compose multiple constraints and generalizes to unseen scenes, objects, and sentences in simulated environments and real-world environments.
  arXiv  Detail & Related papers  (2022-04-11T15:22:43Z)
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents [111.33545170562337]
  We investigate the possibility of grounding high-level tasks, expressed in natural language, to a chosen set of actionable steps. We find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into low-level plans. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions.
  arXiv  Detail & Related papers  (2022-01-18T18:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.