Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation

• URL: http://arxiv.org/abs/2606.23157v1
• Date: Mon, 22 Jun 2026 11:01:56 GMT
• Title: Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation
• Abstract summary: The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution.<n>The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning.
• Score: 0.6927055673104934
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This paper presents a modular training-free framework for zero-shot, language-guided robotic manipulation in semi-structured environments. The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution. To overcome the spatial ambiguity and semantic hallucinations inherent in standard Vision-Language Models (VLMs), the perception module employs FastSAM and Set-of-Mark (SoM) prompting to dynamically generate grounded, alphanumeric visual anchors. The same foundation model then operates purely as a Large Language Model (LLM) to act as a semantic router, translating unconstrained human directives into verifiable, reconfigurable configurations. Finally, these configurations are dynamically parsed by a Task Orchestrator into MoveIt Task Constructor (MTC) to generate collision-free trajectories. The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning, achieving a 62% end-to-end task success rate across both scenarios, demonstrating its capacity to reliably execute complex physical actions without domain-specific training or manual coordinate programming.

Related papers

• KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding [58.541098216700796]
  evaluating the physical consistency of embodied world models is a critical open challenge.<n>Existing frameworks almost exclusively rely on Inverse Dynamics Models(IDMs) for action extraction.<n>We present KineBench, an IDM-free closed-loop benchmark for EWMs.
  arXiv  Detail & Related papers  (2026-07-22T08:04:17Z)
• Dynamics-Aware Meta-Imitation for Generalization to Unseen Robotic Manipulation [17.382238044141783]
  We propose a framework for learning skills from demonstrations for robots.<n>By integrating meta-learning to construct a shared skill space, DAMI equips agents for rapid adaptation to novel tasks.<n>Our approach outperforms state-of-the-art baselines regarding direct inference on seen tasks and adaptation to unseen tasks.
  arXiv  Detail & Related papers  (2026-07-17T11:50:03Z)
• AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learning [52.47988016454267]
  Vision-Language-Action (VLA) models excel at end-to-end robotic manipulation but struggle with out-of-distribution (OOD) generalization.<n>We identify two mutually reinforcing modes: emphtrajectory overfitting, where models overfit to holistic trajectory patterns rather than compositional sub-skill semantics; and emphperceptual shortcut, where action tokens over-rely on wrist-view textures at the expense of global spatial grounding.<n>We introduce textbfAC-VLA, a plug-and-play Action Compositional learning
  arXiv  Detail & Related papers  (2026-07-17T07:51:03Z)
• In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics [109.99871326497333]
  In-Context Model Predictive Generation (ICMPG) is a framework that integrates language-model planning with inference-time physical feedback.<n>ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules.
  arXiv  Detail & Related papers  (2026-06-25T12:50:33Z)
• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning [99.49739831030388]
  We study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning.<n>We propose SpatialClaw, a training-free framework for spatial reasoning that adopts code as the action interface.
  arXiv  Detail & Related papers  (2026-06-11T17:59:36Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• STABLE: Simulation-Ready Tabletop Layout Generation via a Semantics-Physics Dual System [76.3920413336862]
  We present STABLE, a semantics-physics dual-system tailored for simulation-ready tabletop scene generation.<n>STABLE consists of two complementary modules: a Semantic Reasoner and a Physics Corrector.<n> Experiments demonstrate that STABLE successfully generates simulation-ready tabletop scenes that strictly conform to task instructions.
  arXiv  Detail & Related papers  (2026-05-15T16:18:42Z)
• Plan in Sandbox, Navigate in Open Worlds: Learning Physics-Grounded Abstracted Experience for Embodied Navigation [63.11032720821731]
  textitSAGE is a framework that enables agents to learn within a physics-grounded semantic abstraction rather than a photorealistic simulation.<n>We demonstrate that textitSAGE significantly improves planner-assisted embodied navigation, achieving a 53.21% LLM-Match Success Rate on A-EQA.
  arXiv  Detail & Related papers  (2026-05-11T07:34:30Z)
• UniManip: General-Purpose Zero-Shot Robotic Manipulation with Agentic Operational Graph [23.060488218180936]
  We present UniManip, a framework grounded in a Bi-level Agentic Operational Graph (AOG)<n>By coupling a high-level Agentic Layer for task orchestration with a low-level Scene Layer for dynamic state representation, the system continuously aligns abstract planning with geometric constraints.<n>Experiments validate the system's robust zero-shot capability on unseen objects and tasks, demonstrating a 22.5% and 25.0% higher success rate compared to state-of-the-art VLA and hierarchical baselines.
  arXiv  Detail & Related papers  (2026-02-13T16:47:26Z)
• Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation [70.8381970762877]
  Vision-Language Models (VLMs) have demonstrated remarkable capabilities in semantic reasoning and task planning.<n>We introduce GRACE, a novel framework that grounds VLM-based reasoning through executable analytic concepts.<n>G GRACE provides a unified and interpretable interface between high-level instruction understanding and low-level robot control.
  arXiv  Detail & Related papers  (2025-10-09T09:08:33Z)
• Hierarchical Language Models for Semantic Navigation and Manipulation in an Aerial-Ground Robotic System [8.88014241557266]
  Heterogeneous multirobot systems show great potential in complex tasks requiring coordinated hybrid cooperation.<n>Existing methods that rely on static or task-specific models often lack generalizability across diverse tasks and dynamic environments.<n>We propose a hierarchical multimodal framework that integrates a prompted large language model (LLM) with a fine-tuned vision-language model (VLM)
  arXiv  Detail & Related papers  (2025-06-05T13:27:41Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models [38.503337052122234]
  Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation. We aim to synthesize robot trajectories for a variety of manipulation tasks given an open-set of instructions and an open-set of objects. We demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions.
  arXiv  Detail & Related papers  (2023-07-12T07:40:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.