ROSClaw: A Hierarchical Semantic-Physical Framework for Heterogeneous Multi-Agent Collaboration

• URL: http://arxiv.org/abs/2604.04664v1
• Date: Mon, 06 Apr 2026 13:16:24 GMT
• Title: ROSClaw: A Hierarchical Semantic-Physical Framework for Heterogeneous Multi-Agent Collaboration
• Abstract summary: ROSClaw is an agent framework for heterogeneous robots that integrates policy learning and task execution within a unified vision-language model (VLM) controller.<n>By establishing an autonomous closedloop framework, ROSClaw minimizes the reliance on robot-specific development.
• Score: 9.434833852972353
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The integration of large language models (LLMs) with embodied agents has improved high-level reasoning capabilities; however, a critical gap remains between semantic understanding and physical execution. While vision-language-action (VLA) and vision-language-navigation (VLN) systems enable robots to perform manipulation and navigation tasks from natural language instructions, they still struggle with long-horizon sequential and temporally structured tasks. Existing frameworks typically adopt modular pipelines for data collection, skill training, and policy deployment, resulting in high costs in experimental validation and policy optimization. To address these limitations, we propose ROSClaw, an agent framework for heterogeneous robots that integrates policy learning and task execution within a unified vision-language model (VLM) controller. The framework leverages e-URDF representations of heterogeneous robots as physical constraints to construct a sim-to-real topological mapping, enabling real-time access to the physical states of both simulated and real-world agents. We further incorporate a data collection and state accumulation mechanism that stores robot states, multimodal observations, and execution trajectories during real-world execution, enabling subsequent iterative policy optimization. During deployment, a unified agent maintains semantic continuity between reasoning and execution, and dynamically assigns task-specific control to different agents, thereby improving robustness in multi-policy execution. By establishing an autonomous closed-loop framework, ROSClaw minimizes the reliance on robot-specific development workflows. The framework supports hardware-level validation, automated generation of SDK-level control programs, and tool-based execution, enabling rapid cross-platform transfer and continual improvement of robotic skills. Ours project page: https://www.rosclaw.io/.

Related papers

• Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework [0.0]
  Large Language Models (LLMs) have enabled more natural human-robot interaction, but open-source models exhibit unstable long-horizon reasoning.<n>This paper presents an enhanced ROS-Agent based architecture that improves task reliability and execution efficiency.
  arXiv  Detail & Related papers  (2026-09-11T10:00:49Z)
• MobileVLA-R1 2.0: RL-Enhanced Reasoning for Mobile Robot Control [63.85634404397772]
  MobileVLA-R1 2.0 is an RL-enhanced vision-language-action framework for mobile robots.<n>It learns multi-granularity reasoning over embodied trajectories through supervised Chain-of-Thought (CoT) alignment and reinforcement learning.<n>We conduct evaluations on language-guided navigation, quadruped control, and humanoid mobile manipulation, covering VLN-CE, QUARD, and real-world deployments on Unitree Go2 and G1 robots.
  arXiv  Detail & Related papers  (2026-09-05T20:31:14Z)
• Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy [79.09714980186278]
  Building persistent embodied agents in unstructured environments requires unified orchestration of heterogeneous tools.<n>We present OmniAct, a framework integrating a multimodal semantic planner for skill routing across unified action spaces.<n>It achieves consistent improvements in end-to-end success across all complexity levels.
  arXiv  Detail & Related papers  (2026-06-25T16:36:35Z)
• Closed-Loop Verbal Reinforcement Learning for Task-Level Robotic Planning [1.6641413441634143]
  We propose a new Verbal Reinforcement Learning framework for interpretable task-level planning in mobile robotic systems.<n>The framework follows a closed-loop architecture that enables iterative policy improvement through interaction with the physical environment.<n>We validate the proposed framework on a real mobile robot performing a multi-stage manipulation and navigation task under execution uncertainty.
  arXiv  Detail & Related papers  (2026-03-23T16:28:36Z)
• RoboNeuron: A Modular Framework Linking Foundation Models and ROS for Embodied AI [13.74517467087138]
  RoboNeuron is a universal deployment framework for embodied intelligence.<n>It is the first framework to deeply integrate the cognitive capabilities of Large Language Models (LLMs) and Vision-Language-Action (VLA) models with the real-time execution backbone of the Robot Operating System (ROS)
  arXiv  Detail & Related papers  (2025-12-11T07:58:19Z)
• Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation [70.8381970762877]
  Vision-Language Models (VLMs) have demonstrated remarkable capabilities in semantic reasoning and task planning.<n>We introduce GRACE, a novel framework that grounds VLM-based reasoning through executable analytic concepts.<n>G GRACE provides a unified and interpretable interface between high-level instruction understanding and low-level robot control.
  arXiv  Detail & Related papers  (2025-10-09T09:08:33Z)
• State and Memory is All You Need for Robust and Reliable AI Agents [29.259008600842517]
  Large language models (LLMs) have enabled powerful advances in natural language understanding and generation.<n>Yet their application to complex, real-world scientific remain limited by challenges in memory, planning, and tool integration.<n>Here, we introduce SciBORG, a modular agentic framework that allows LLM-based agents to autonomously plan, reason, and achieve robust and reliable domain-specific task execution.
  arXiv  Detail & Related papers  (2025-06-30T02:02:35Z)
• From Virtual Agents to Robot Teams: A Multi-Robot Framework Evaluation in High-Stakes Healthcare Context [2.016235597066821]
  Current frameworks treat agents as conceptual task executors rather than physically embodied entities.<n>We propose three design guidelines emphasizing process transparency, proactive failure recovery, and contextual grounding.<n>Our work informs the development of more resilient and robust multi-agent robotic systems.
  arXiv  Detail & Related papers  (2025-06-04T04:05:38Z)
• RoBridge: A Hierarchical Architecture Bridging Cognition and Execution for General Robotic Manipulation [90.81956345363355]
  RoBridge is a hierarchical intelligent architecture for general robotic manipulation.<n>It consists of a high-level cognitive planner (HCP) based on a large-scale pre-trained vision-language model (VLM)<n>It unleashes the procedural skill of reinforcement learning, effectively bridging the gap between cognition and execution.
  arXiv  Detail & Related papers  (2025-05-03T06:17:18Z)
• Embodied Long Horizon Manipulation with Closed-loop Code Generation and Incremental Few-shot Adaptation [12.077740860502878]
  Embodied long-horizon manipulation requires robotic systems to process multimodal inputs-such as vision and natural language-and translate them into executable actions.<n>Recent methods have explored using large language models (LLMs) as high-level planners that decompose tasks into subtasks using natural language and guide pretrained low-level controllers.<n>Our framework achieves state-of-the-art performance on 30+ diverse seen and unseen long-horizon tasks across LoHoRavens, CALVIN, Franka Kitchen, and cluttered real-world settings.
  arXiv  Detail & Related papers  (2025-03-27T20:32:58Z)
• ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning [74.58666091522198]
  We present a framework for intuitive robot programming by non-experts. We leverage natural language prompts and contextual information from the Robot Operating System (ROS) Our system integrates large language models (LLMs), enabling non-experts to articulate task requirements to the system through a chat interface.
  arXiv  Detail & Related papers  (2024-06-28T08:28:38Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.