Agentic AI for Robot Control: Flexible but still Fragile

• URL: http://arxiv.org/abs/2602.13081v1
• Date: Fri, 13 Feb 2026 16:43:34 GMT
• Title: Agentic AI for Robot Control: Flexible but still Fragile
• Abstract summary: We present an agentic control system in which a reasoning-capable language model plans and executes tasks.<n>We deploy the system on two physical robot platforms in two settings: (i) tabletop grasping, placement, and box insertion in indoor mobile manipulation (Mobipick) and (ii) autonomous agricultural navigation and sensing (Valdemar)
• Score: 32.43712661386947
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work leverages the capabilities and commonsense priors of generative models for robot control. In this paper, we present an agentic control system in which a reasoning-capable language model plans and executes tasks by selecting and invoking robot skills within an iterative planner and executor loop. We deploy the system on two physical robot platforms in two settings: (i) tabletop grasping, placement, and box insertion in indoor mobile manipulation (Mobipick) and (ii) autonomous agricultural navigation and sensing (Valdemar). Both settings involve uncertainty, partial observability, sensor noise, and ambiguous natural-language commands. The system exposes structured introspection of its planning and decision process, reacts to exogenous events via explicit event checks, and supports operator interventions that modify or redirect ongoing execution. Across both platforms, our proof-of-concept experiments reveal substantial fragility, including non-deterministic suboptimal behavior, instruction-following errors, and high sensitivity to prompt specification. At the same time, the architecture is flexible: transfer to a different robot and task domain largely required updating the system prompt (domain model, affordances, and action catalogue) and re-binding the same tool interface to the platform-specific skill API.

Related papers

• Learning Physical Interaction: A Survey of Tactile- and Force-aware Robot Learning [128.76135832773102]
  Physically grounded robot intelligence requires robots to perceive, reason about, and regulate their interactions with the physical world.<n>Recent robot learning methods have made substantial progress by integrating force, tactile, vision, language, and proprioceptive sensing into learned manipulation policies.<n>Despite these advances, existing surveys have not explicitly reviewed force- and tactile-aware robot learning from a unified perspective.
  arXiv  Detail & Related papers  (2026-08-02T14:22:01Z)
• RoboWits: Unexpected Challenges for Robotic Creative Problem Solving [58.58727722988084]
  We introduce RoboWits, a bi-manual robotic benchmark designed to evaluate cognitive reasoning, creative tool use, and robustness to unexpected conditions.<n>Using the pipeline, we curated 30 diverse seed tasks and 208 tasks with mutations and graded difficulty across geometry, material, and assembly-based reasoning.<n>Our results reveal a significant performance gap: while pre-trained VLAs exhibit preliminary success on seed tasks after single-task fine-tuning, they struggle to perform on mutated tasks.
  arXiv  Detail & Related papers  (2026-05-28T17:57:15Z)
• RACAS: Controlling Diverse Robots With a Single Agentic System [26.695876900159373]
  We introduce RACAS (Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture in which three modules communicate exclusively through natural language to provide closed-loop robot control.<n> RACAS requires only a natural language description of the robot, a definition of available actions, and a task specification; no source code, model weights, or reward functions need to be modified to move between platforms.<n>We evaluate RACAS on several tasks using a wheeled ground robot, a recently published novel multi-jointed robotic limb, and an underwater vehicle. RACAS consistently solved all assigned tasks across these radically different platforms, demonstrating the potential
  arXiv  Detail & Related papers  (2026-03-05T19:25:01Z)
• Robot guide with multi-agent control and automatic scenario generation with LLM [0.0]
  The work describes the development of a hybrid control architecture for an anthropomorphic tour guide robot.<n>The proposed approach aims to overcome the limitations of traditional systems, which rely on manual tuning of behavior scenarios.
  arXiv  Detail & Related papers  (2025-09-12T14:59:04Z)
• OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis [70.39500621448383]
  Open-world mobile manipulation task remains a challenge due to the need for generalization to open-ended instructions and environments.<n>We propose a novel multi-modal agent architecture that maintains multi-view scene frames and agent states for decision-making and controls the robot by function calling.<n>We highlight our fine-tuned OWMM-VLM as the first dedicated foundation model for mobile manipulators with global scene understanding, robot state tracking, and multi-modal action generation in a unified model.
  arXiv  Detail & Related papers  (2025-06-04T17:57:44Z)
• Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models [81.55156507635286]
  Legged robots are physically capable of navigating a diverse variety of environments and overcoming a wide range of obstructions. Current learning methods often struggle with generalization to the long tail of unexpected situations without heavy human supervision. We propose a system, VLM-Predictive Control (VLM-PC), combining two key components that we find to be crucial for eliciting on-the-fly, adaptive behavior selection.
  arXiv  Detail & Related papers  (2024-07-02T21:00:30Z)
• ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning [74.58666091522198]
  We present a framework for intuitive robot programming by non-experts. We leverage natural language prompts and contextual information from the Robot Operating System (ROS) Our system integrates large language models (LLMs), enabling non-experts to articulate task requirements to the system through a chat interface.
  arXiv  Detail & Related papers  (2024-06-28T08:28:38Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• Language-Conditioned Robotic Manipulation with Fast and Slow Thinking [30.36976573047372]
  We introduce Robotics with Fast and Slow Thinking (RFST), a framework that mimics human cognitive architecture to classify tasks. RFST consists of two key components: 1) an instruction discriminator to determine which system should be activated based on the current user instruction, and 2) a slow-thinking system that is comprised of a fine-tuned vision language model aligned with the policy networks. Our results, both in simulation and real-world scenarios, confirm that our approach adeptly manages intricate tasks that demand intent recognition and reasoning.
  arXiv  Detail & Related papers  (2024-01-08T19:00:32Z)
• SkiROS2: A skill-based Robot Control Platform for ROS [1.4502611532302039]
  We introduce SkiROS2, a skill-based robot control platform on top of ROS. SkiROS2 proposes a layered, hybrid control structure for automated task planning, and reactive execution. We relate SkiROS2 to the field and outline three example use cases that cover task planning, reasoning, multisensory input, integration in a manufacturing execution system and reinforcement learning.
  arXiv  Detail & Related papers  (2023-06-29T15:25:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.