Harness Engineering for Physical AI: Robot Middleware Is the Harness Layer

• URL: http://arxiv.org/abs/2606.09416v1
• Date: Mon, 08 Jun 2026 12:29:54 GMT
• Title: Harness Engineering for Physical AI: Robot Middleware Is the Harness Layer
• Abstract summary: A Physical AI harness must mediate at control, computing, and communication simultaneously.<n>We name this enforcement as three functions: Projection gates each output at emission, Isolation bounds the model's execution and transmission slot, and Transfer falls back to a verified baseline when checks fail.<n>We sketch this as a ROS 2 Profile, a deployment artifact that carries an AI model's declared output region, inference budget, and operating regime.
• Score: 14.262395972290564
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robot middleware faces a new role in the era of Physical AI. Learned policies, planners, and vision-language-action (VLA) models now enter deployed robots as causal participants on the control path, but the layer that integrates them with timing, scheduling, and network has not been named. Recent language-agent work names this layer the harness, the external system that mediates tools, manages state, bounds resources, and records execution. The robotics community has not yet adopted this framing, and we propose that robot middleware is that harness. A Physical AI harness differs from a software harness in where it intervenes. A software harness mediates at tool-call boundaries. A Physical AI harness must mediate at control, computing, and communication simultaneously, because a learned policy's output crosses all three: its commands shift the trajectory, its inference time shifts the schedule, and its payload shifts the bandwidth. Robot middleware is the lowest robot-stack layer with mediating abstractions over all three, so it is best positioned to compose their enforcement. It already provides most of what a harness needs but lacks the enforcement for an AI model. We name this missing enforcement as three functions: Projection gates each output at emission, Isolation bounds the model's execution and transmission slot, and Transfer falls back to a verified baseline when checks fail. Each appears today as hand-built application code in deployed robot systems, built on surfaces robot middleware already provides. Robot middleware should host them not as the best single-axis enforcer but as the layer that composes all three. We sketch this as a ROS 2 Harness Profile, a deployment artifact that carries an AI model's declared output region, inference budget, and operating regime while the middleware enforces them across ROS 2, DDS, and Zenoh.

Related papers

• Physical Agentic AI: An Architecture for Orchestrating a Robot Crew with LLMs [6.2312877626161205]
  This paper introduces Physical Agentic AI, a framework for skill-grounded robot agent orchestration.<n>A Robot Orchestration layer exposes the skill library, robot state, named locations, and workflow contracts to a non-actuating Mission Planner.<n>We evaluate on a drone-UGV search-and-dispatch mission, where every mission in every condition is executed live in Gazebo.
  arXiv  Detail & Related papers  (2026-08-23T23:35:31Z)
• ORCESTRA: VLM-driven Visual Robot programming in Mixed Reality [2.1645011609137295]
  ORCESTRA is a mixed-reality system for programming robot digital twins through no-code waypoint teaching and language-guided control.<n>In a passthrough mixed-reality workspace, users place robot twins on real surfaces, teach trajectories, save robot-relative episodes, or issue spoken/typed commands that a vision-language model converts into structured digital-twin plans.
  arXiv  Detail & Related papers  (2026-08-01T17:12:01Z)
• VIA: Visual Interface Agent for Robot Control [34.69686658431114]
  VIA (Visual Interface Agent for robot control) is a framework that recasts robot control as an agentic task.<n>It drives a manipulator through a browser-based 3D interface by taking screenshots, issuing intuitive commands, observing the outcome, and adjusting.<n>It achieves 96.7% success on three LIBERO-Goal tasks and 100% on a long-horizon rainbow assembly task.
  arXiv  Detail & Related papers  (2026-07-13T05:52:12Z)
• WSA$_1$: a 3D-Centric World-Spatial-Action Model for Generalizable Robot Control [85.21175469425172]
  Recent advances in embodied AI have established robot foundation models (RFMs) as the dominant approach for generalist robotic systems to date.<n>We present WSA$_1$, a novel RFM built upon proposed 3D-Centric World-Spatial-Action modeling paradigm.<n>It not only learns 3D world-aware visual thought for future robot behaviors, but also models mutual constraints between 3D world state transitions and robotic actions to enhance behavior generalization.
  arXiv  Detail & Related papers  (2026-07-04T16:26:39Z)
• RACAS: Controlling Diverse Robots With a Single Agentic System [26.695876900159373]
  We introduce RACAS (Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture in which three modules communicate exclusively through natural language to provide closed-loop robot control.<n> RACAS requires only a natural language description of the robot, a definition of available actions, and a task specification; no source code, model weights, or reward functions need to be modified to move between platforms.<n>We evaluate RACAS on several tasks using a wheeled ground robot, a recently published novel multi-jointed robotic limb, and an underwater vehicle. RACAS consistently solved all assigned tasks across these radically different platforms, demonstrating the potential
  arXiv  Detail & Related papers  (2026-03-05T19:25:01Z)
• GR00T N1: An Open Foundation Model for Generalist Humanoid Robots [133.23509142762356]
  General-purpose robots need a versatile body and an intelligent mind.<n>Recent advancements in humanoid robots have shown great promise as a hardware platform for building generalist autonomy.<n>We introduce GR00T N1, an open foundation model for humanoid robots.
  arXiv  Detail & Related papers  (2025-03-18T21:06:21Z)
• Jailbreaking LLM-Controlled Robots [82.04590367171932]
  Large language models (LLMs) have revolutionized the field of robotics by enabling contextual reasoning and intuitive human-robot interaction. LLMs are vulnerable to jailbreaking attacks, wherein malicious prompters elicit harmful text by bypassing LLM safety guardrails. We introduce RoboPAIR, the first algorithm designed to jailbreak LLM-controlled robots.
  arXiv  Detail & Related papers  (2024-10-17T15:55:36Z)
• Generalizable Humanoid Manipulation with 3D Diffusion Policies [66.78220965526732]
  We build a real-world robotic system to address the problem of autonomous manipulation by humanoid robots.<n>Our system is mainly an integration of 1) a whole-upper-body robotic teleoperation system to acquire human-like robot data, and 2) a 25-DoF humanoid robot platform with a height-adjustable cart and a 3D LiDAR sensor.<n>We show that using only data collected in one scene and with only onboard computing, a full-sized humanoid robot can autonomously perform skills in diverse real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-14T17:59:00Z)
• CyberCortex.AI: An AI-based Operating System for Autonomous Robotics and Complex Automation [0.0]
  We introduce CyberCortex AI, a robotics OS designed to enable heterogeneous AI-based robotics and complex automation applications. CyberCortex AI is a decentralized distributed OS which enables robots to talk to each other, as well as to High Performance Computers in the cloud. Sensory and control data from the robots is streamed towards HPC systems with the purpose of training AI algorithms, which are afterwards deployed on the robots.
  arXiv  Detail & Related papers  (2024-09-02T13:14:50Z)
• GRUtopia: Dream General Robots in a City at Scale [65.08318324604116]
  This paper introduces project GRUtopia, the first simulated interactive 3D society designed for various robots. GRScenes includes 100k interactive, finely annotated scenes, which can be freely combined into city-scale environments. GRResidents is a Large Language Model (LLM) driven Non-Player Character (NPC) system that is responsible for social interaction.
  arXiv  Detail & Related papers  (2024-07-15T17:40:46Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.