Retrieval-Augmented Robots via Retrieve-Reason-Act

• URL: http://arxiv.org/abs/2603.02688v1
• Date: Tue, 03 Mar 2026 07:32:15 GMT
• Title: Retrieval-Augmented Robots via Retrieve-Reason-Act
• Abstract summary: We argue that robots must evolve from passive executors into active Information Retrieval users.<n>In strictly zero-shot settings where no prior demonstrations exist, robots face a critical information gap.<n>We define the paradigm as Retrieval-Augmented Robotics (RAR), empowering the robot with the information-seeking capability.
• Score: 5.93914911947526
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To achieve general-purpose utility, we argue that robots must evolve from passive executors into active Information Retrieval users. In strictly zero-shot settings where no prior demonstrations exist, robots face a critical information gap, such as the exact sequence required to assemble a complex furniture kit, that cannot be satisfied by internal parametric knowledge (common sense) or past internal memory. While recent robotic works attempt to use search before action, they primarily focus on retrieving past kinematic trajectories (analogous to searching internal memory) or text-based safety rules (searching for constraints). These approaches fail to address the core information need of active task construction: acquiring unseen procedural knowledge from external, unstructured documentation. In this paper, we define the paradigm as Retrieval-Augmented Robotics (RAR), empowering the robot with the information-seeking capability that bridges the gap between visual documentation and physical actuation. We formulate the task execution as an iterative Retrieve-Reason-Act loop: the robot or embodied agent actively retrieves relevant visual procedural manuals from an unstructured corpus, grounds the abstract 2D diagrams to 3D physical parts via cross-modal alignment, and synthesizes executable plans. We validate this paradigm on a challenging long-horizon assembly benchmark. Our experiments demonstrate that grounding robotic planning in retrieved visual documents significantly outperforms baselines relying on zero-shot reasoning or few-shot example retrieval. This work establishes the basis of RAR, extending the scope of Information Retrieval from answering user queries to driving embodied physical actions.

Related papers

• DocOS: Towards Proactive Document-Guided Actions in GUI Agents [54.27655693145045]
  textbfDocOS is a benchmark designed to assess document-guided problem solving in fully interactive environments.<n>Experiments reveal that agents struggle to reliably locate relevant information during proactive search and frequently fail to faithfully ground retrieved instructions into precise actions.
  arXiv  Detail & Related papers  (2026-05-18T08:36:04Z)
• EmbodiedLGR: Integrating Lightweight Graph Representation and Retrieval for Semantic-Spatial Memory in Robotic Agents [7.142235510048155]
  EmbodiedLGR-Agent is a visual-language model (VLM)-driven agent architecture that constructs dense and efficient representations of robot operating environments.<n>EmbodiedLGR-Agent is evaluated on the popular NaVQA dataset, achieving state-of-the-art performance in inference and querying times for embodied agents.
  arXiv  Detail & Related papers  (2026-04-20T13:45:54Z)
• ROSER: Few-Shot Robotic Sequence Retrieval for Scalable Robot Learning [40.51327348071904]
  A critical bottleneck in robot learning is the scarcity of task-labeled, segmented training data.<n>We introduce ROSER, a lightweight few-shot retrieval framework that learns task-agnostic metric spaces over temporal windows.<n>Our experiments demonstrate that ROSER consistently outperforms all prior methods in both accuracy and efficiency.
  arXiv  Detail & Related papers  (2026-03-02T05:37:33Z)
• Generalizable Geometric Prior and Recurrent Spiking Feature Learning for Humanoid Robot Manipulation [90.90219129619344]
  This paper presents a novel R-prior-S, Recurrent Geometric-priormodal Policy with Spiking features.<n>To ground high-level reasoning in physical reality, we leverage lightweight 2D geometric inductive biases.<n>For the data efficiency issue in robotic action generation, we introduce a Recursive Adaptive Spiking Network.
  arXiv  Detail & Related papers  (2026-01-13T23:36:30Z)
• Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulation [20.373596661083152]
  Affordance RAG is a zero-shot hierarchical multimodal retrieval framework that constructs Affordance-Aware Embodied Memory from pre-explored images.<n>Our method outperformed existing approaches in retrieval performance for mobile manipulation instruction in large-scale indoor environments.
  arXiv  Detail & Related papers  (2025-12-22T02:55:25Z)
• Is Single-View Mesh Reconstruction Ready for Robotics? [78.14584238127338]
  We evaluate single-view mesh reconstruction models for their potential in enabling instant digital twin creation for real-time planning and dynamics prediction using physics simulators for robotic manipulation.<n>Our findings highlight critical gaps between computer vision advances and robotics needs, guiding future research at this intersection.
  arXiv  Detail & Related papers  (2025-05-23T14:35:56Z)
• Learning Object Properties Using Robot Proprioception via Differentiable Robot-Object Interaction [52.12746368727368]
  Differentiable simulation has become a powerful tool for system identification.<n>Our approach calibrates object properties by using information from the robot, without relying on data from the object itself.<n>We demonstrate the effectiveness of our method on a low-cost robotic platform.
  arXiv  Detail & Related papers  (2024-10-04T20:48:38Z)
• Tiny Robotics Dataset and Benchmark for Continual Object Detection [6.4036245876073234]
  We introduce a novel benchmark to evaluate the continual learning capabilities of object detection systems in tiny robotic platforms.<n>Our results highlight key challenges in developing robust and efficient continual learning strategies for object detectors in tiny robotics.
  arXiv  Detail & Related papers  (2024-09-24T16:21:27Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• SEAL: Semantic Frame Execution And Localization for Perceiving Afforded Robot Actions [5.522839151632667]
  We extend the semantic frame representation for robot manipulation actions and introduce the problem of Semantic Frame Execution And Localization for Perceiving Afforded Robot Actions (SEAL) as a graphical model. For the SEAL problem, we describe our nonparametric Semantic Frame Mapping (SeFM) algorithm for maintaining belief over a finite set of semantic frames as the locations of actions afforded to the robot.
  arXiv  Detail & Related papers  (2023-03-24T15:25:41Z)
• Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? [54.442692221567796]
  Task specification is critical for engagement of non-expert end-users and adoption of personalized robots. A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene. In this work, we explore alternate and more general forms of goal specification that are expected to be easier for humans to specify and use.
  arXiv  Detail & Related papers  (2022-04-23T19:39:49Z)
• Memory-based gaze prediction in deep imitation learning for robot manipulation [2.857551605623957]
  The proposed algorithm uses a Transformer-based self-attention architecture for the gaze estimation based on sequential data to implement memory. The proposed method was evaluated with a real robot multi-object manipulation task that requires memory of the previous states.
  arXiv  Detail & Related papers  (2022-02-10T07:30:08Z)
• Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents [61.36681529571202]
  We describe a new concept for reproducible robotics research that integrates development and benchmarking. One of the central components of this setup is the Duckietown Autolab, a standardized setup that is itself relatively low-cost and reproducible. We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.
  arXiv  Detail & Related papers  (2020-09-09T15:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.