Experience is the Best Teacher: Grounding VLMs for Robotics through Self-Generated Memory

• URL: http://arxiv.org/abs/2507.16713v1
• Date: Tue, 22 Jul 2025 15:48:49 GMT
• Title: Experience is the Best Teacher: Grounding VLMs for Robotics through Self-Generated Memory
• Abstract summary: This paper presents a framework that grounds vision-language models (VLMs) to robots by building a self-generated memory of real-world experiences.<n>In ExpTeach, the VLM autonomously plans actions, verifies outcomes, reflects on failures, and adapts robot behaviors in a closed loop.<n>In experiments, we show that reflection improves success rates from 36% to 84% on four challenging robotic tasks.
• Score: 16.53566194731369
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) have been widely adopted in robotics to enable autonomous planning. However, grounding VLMs, originally trained on internet data, to diverse real-world robots remains a challenge. This paper presents ExpTeach, a framework that grounds VLMs to physical robots by building a self-generated memory of real-world experiences. In ExpTeach, the VLM autonomously plans actions, verifies outcomes, reflects on failures, and adapts robot behaviors in a closed loop. The self-generated experiences during this process are then summarized into a long-term memory, enabling retrieval of learned knowledge to guide future tasks via retrieval-augmented generation (RAG). Additionally, ExpTeach enhances the spatial understanding of VLMs with an on-demand image annotation module. In experiments, we show that reflection improves success rates from 36% to 84% on four challenging robotic tasks and observe the emergence of intelligent object interactions, including creative tool use. Across extensive tests on 12 real-world scenarios (including eight unseen ones), we find that grounding with long-term memory boosts single-trial success rates from 22% to 80%, demonstrating the effectiveness and generalizability of ExpTeach.

Related papers

• MessyMem: Learning-from-Doing Memory for Mobile Manipulation [12.915040804075282]
  MessyMem is a persistent memory system that enables mobile manipulators to learn from experience and reuse that knowledge across future tasks.<n>We present MessyMem, a persistent memory system that enables mobile manipulators to learn from experience and reuse that knowledge across future tasks.
  arXiv  Detail & Related papers  (2026-09-14T17:56:49Z)
• MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models [80.70528162709276]
  Temporal modeling is essential for robotic manipulation, as effective control requires memory of past interactions and imagination of future states.<n>We propose MemoryVLA++, a full temporal modeling framework that equips VLA models with memory and imagination for robotic manipulation.<n>Our method achieves strong performance across Libero, SimplerEnv, Mikasa-Robo, Calvin, Libero-Plus, and diverse real-robot tasks.
  arXiv  Detail & Related papers  (2026-06-08T17:59:53Z)
• VER: Vision Expert Transformer for Robot Learning via Foundation Distillation and Dynamic Routing [89.48383845451717]
  We propose VER, a Vision Expert transformer for Robot learning.<n>During pretraining, VER distills multiple VFMs into a vision expert library.<n>It then fine-tunes only a lightweight routing network to dynamically select task-relevant experts.
  arXiv  Detail & Related papers  (2025-10-06T18:00:43Z)
• Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets [15.349107660229356]
  Vision-Language Models (VLMs) acquire real-world knowledge and general reasoning ability through Internet-scale image-text corpora.<n>We present Robo2VLM, a Visual Question Answering (VQA) dataset generation framework for VLMs.<n>We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710 questions covering 463 distinct scenes and 3,396 robotic manipulation tasks from 176k real robot trajectories.
  arXiv  Detail & Related papers  (2025-05-21T13:42:52Z)
• Efficient Reinforcement Learning by Guiding Generalist World Models with Non-Curated Data [32.7248232143849]
  Leveraging offline data is a promising way to improve the sample efficiency of online reinforcement learning (RL)<n>This paper expands the pool of usable data for offline-to-online RL by leveraging abundant non-curated data that is reward-free, of mixed quality, and collected across multiple embodiments.
  arXiv  Detail & Related papers  (2025-02-26T20:34:29Z)
• Towards General Purpose Robots at Scale: Lifelong Learning and Learning to Use Memory [0.0]
  This thesis focuses on addressing two key challenges for robots operating over long time horizons: memory and lifelong learning.<n>First, we introduce t-DGR, a trajectory-based deep generative replay method that achieves state-of-the-art performance on Continual World benchmarks.<n>Second, we develop a framework that leverages human demonstrations to teach agents effective memory utilization.
  arXiv  Detail & Related papers  (2024-12-28T21:13:48Z)
• Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models [81.55156507635286]
  Legged robots are physically capable of navigating a diverse variety of environments and overcoming a wide range of obstructions. Current learning methods often struggle with generalization to the long tail of unexpected situations without heavy human supervision. We propose a system, VLM-Predictive Control (VLM-PC), combining two key components that we find to be crucial for eliciting on-the-fly, adaptive behavior selection.
  arXiv  Detail & Related papers  (2024-07-02T21:00:30Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations.<n>First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets.<n>We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)
• LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning [50.99807031490589]
  We introduce LLARVA, a model trained with a novel instruction tuning method to unify a range of robotic learning tasks, scenarios, and environments. We generate 8.5M image-visual trace pairs from the Open X-Embodiment dataset in order to pre-train our model. Experiments yield strong performance, demonstrating that LLARVA performs well compared to several contemporary baselines.
  arXiv  Detail & Related papers  (2024-06-17T17:55:29Z)
• Towards Generalist Robot Learning from Internet Video: A Survey [56.621902345314645]
  This survey systematically examines the emerging field of learning from videos (LfV)<n>We first outline essential concepts, including detailing fundamental LfV challenges such as distribution shift and missing action labels in video data.<n>Next, we comprehensively review current methods for extracting knowledge from large-scale internet video, overcoming LfV challenges, and improving robot learning through video-informed training.
  arXiv  Detail & Related papers  (2024-04-30T15:57:41Z)
• Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning [32.045840007623276]
  We introduce Robotic Vision-Language Planning (ViLa), a novel approach for long-horizon robotic planning. ViLa directly integrates perceptual data into its reasoning and planning process. Our evaluation, conducted in both real-robot and simulated environments, demonstrates ViLa's superiority over existing LLM-based planners.
  arXiv  Detail & Related papers  (2023-11-29T17:46:25Z)
• LLM as A Robotic Brain: Unifying Egocentric Memory and Control [77.0899374628474]
  Embodied AI focuses on the study and development of intelligent systems that possess a physical or virtual embodiment (i.e. robots) Memory and control are the two essential parts of an embodied system and usually require separate frameworks to model each of them. We propose a novel framework called LLM-Brain: using Large-scale Language Model as a robotic brain to unify egocentric memory and control.
  arXiv  Detail & Related papers  (2023-04-19T00:08:48Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• SQUIRL: Robust and Efficient Learning from Video Demonstration of Long-Horizon Robotic Manipulation Tasks [8.756012472587601]
  Deep reinforcement learning (RL) can be used to learn complex manipulation tasks. RL requires the robot to collect a large amount of real-world experience. S SQUIRL performs a new but related long-horizon task robustly given only a single video demonstration.
  arXiv  Detail & Related papers  (2020-03-10T20:26:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.