VLM-Guided Experience Replay

• URL: http://arxiv.org/abs/2602.01915v1
• Date: Mon, 02 Feb 2026 10:19:59 GMT
• Title: VLM-Guided Experience Replay
• Abstract summary: We propose to use Vision-Language Models (VLMs) to guide the prioritization of experiences in the replay buffer.<n>Our key idea is to use a frozen, pre-trained VLM as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences.<n>Across scenarios, including game-playing and robotics, agents trained with our proposed prioritization method achieve 11-52% higher average success rates and improve sample efficiency by 19-45% compared to previous approaches.
• Score: 41.08659748023147
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have enabled powerful semantic and multimodal reasoning capabilities, creating new opportunities to enhance sample efficiency, high-level planning, and interpretability in reinforcement learning (RL). While prior work has integrated LLMs and VLMs into various components of RL, the replay buffer, a core component for storing and reusing experiences, remains unexplored. We propose addressing this gap by leveraging VLMs to guide the prioritization of experiences in the replay buffer. Our key idea is to use a frozen, pre-trained VLM (requiring no fine-tuning) as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences. Across scenarios, including game-playing and robotics, spanning both discrete and continuous domains, agents trained with our proposed prioritization method achieve 11-52% higher average success rates and improve sample efficiency by 19-45% compared to previous approaches. https://esharony.me/projects/vlm-rb/

Related papers

• Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning [50.464623632604976]
  In this work, we study RL-based training of vision-language models (VLMs) for long-horizon decision-making in Super Mario Land.<n>We propose an adapted variant of PPO with a lightweight turn-level critic, which substantially improves training stability and sample efficiency.<n>We introduce Odysseus, an open training framework for VLM agents, achieving substantial gains across multiple levels of the game.
  arXiv  Detail & Related papers  (2026-05-01T02:05:56Z)
• Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning [43.63475878891097]
  Reinforcement Learning (RL) has achieved impressive success in post-training Large Language Models (LLMs) and Vision-Language Models (VLMs)<n>These methods discard all collected trajectories after a single gradient update, resulting in poor sample efficiency.<n>We propose Freshness-Aware PER, which addresses this priority staleness problem by augmenting any PER-based priority with a multiplicative exponential age decay.
  arXiv  Detail & Related papers  (2026-04-18T08:51:29Z)
• Retrieval-Augmented LLM Agents: Learning to Learn from Experience [16.248836438253814]
  We study how to train retrieval-augmented LLM agents to leverage retrieved trajectories in-context.<n>We establish a robust supervised fine-tuning (SFT) recipe using LoRA that outperforms several state-of-the-art agent training pipelines.<n>Our results demonstrate that this combined approach significantly improves generalization to unseen tasks.
  arXiv  Detail & Related papers  (2026-03-18T20:45:04Z)
• Action Draft and Verify: A Self-Verifying Framework for Vision-Language-Action Model [31.013109374489442]
  Vision-Language-Action (VLA) models have recently demonstrated strong performance across embodied tasks.<n>We propose Action-Draft-and-Verify (ADV): diffusion action expert drafts multiple candidate action chunks, and the VLM selects one by scoring all candidates in a single forward pass with a perplexity-style metric.<n>Under matched backbones, training data, and action-chunk length, ADV improves success rate by +4.3 points in simulation and +19.7 points in real-world over diffusion-based baseline.
  arXiv  Detail & Related papers  (2026-03-18T09:16:20Z)
• ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation [57.399685080574756]
  Existing MLLM-based VLN methods rely on imitation learning (IL) and often use DAgger for post-training.<n>We propose ActiveVLN, a VLN framework that explicitly enables active exploration through multi-turn RL.<n>Experiments show that ActiveVLN achieves the largest performance gains over IL baselines compared to both DAgger-based and prior RL-based post-training methods.
  arXiv  Detail & Related papers  (2025-09-16T03:31:46Z)
• MaskSearch: A Universal Pre-Training Framework to Enhance Agentic Search Capability [106.35604230971396]
  Recent advancements in Agent techniques enable Large Language Models (LLMs) to autonomously utilize tools for retrieval, planning, and reasoning.<n>To further enhance the universal search capability of agents, we propose a novel pre-training framework, MaskSearch.<n>In the pre-training stage, we introduce the Retrieval Augmented Mask Prediction (RAMP) task, where the model learns to leverage search tools to fill masked spans.<n>After that, the model is trained on downstream tasks to achieve further improvement.
  arXiv  Detail & Related papers  (2025-05-26T17:58:50Z)
• Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning [17.59802090014789]
  We introduce PrefVLM, a framework that integrates Vision-Language Models (VLMs) with selective human feedback.<n>Our method leverages VLMs to generate initial preference labels, which are then filtered to identify uncertain cases for targeted human annotation.<n> Experiments on Meta-World manipulation tasks demonstrate that PrefVLM achieves comparable or superior success rates to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-02-03T18:50:15Z)
• Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation [109.5893580175657]
  In recent times, the standard practice for developing MLLMs is to feed features from vision encoder(s) into the LLM and train with natural language supervision.<n>This approach often causes models to lean towards language comprehension and undermine the rich visual perception signals present in the data.<n>We propose VisPer-LM, the first approach that infuses visual perception knowledge from expert vision encoders into the LLM's hidden representations.
  arXiv  Detail & Related papers  (2024-12-12T18:55:18Z)
• Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training [48.455597568212944]
  We present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts structure.<n>In particular, EViP is designed as a progressive learning process for visual experts, which aims to fully exploit the visual knowledge from noisy data to high-quality data.
  arXiv  Detail & Related papers  (2024-10-10T17:59:22Z)
• Sparse Rewards Can Self-Train Dialogue Agents [22.799506097310008]
  We introduce a novel self-improvement paradigm that empowers LLM agents to autonomously enhance their performance without external human feedback.<n>We present ToolWOZ, a sparse reward tool-calling simulation environment derived from MultiWOZ.<n>We demonstrate that models trained with JOSH, both small and frontier, significantly improve tool-based interactions while preserving general model capabilities across diverse benchmarks.
  arXiv  Detail & Related papers  (2024-09-06T21:00:57Z)
• Traj-LLM: A New Exploration for Empowering Trajectory Prediction with Pre-trained Large Language Models [12.687494201105066]
  This paper proposes Traj-LLM, the first to investigate the potential of using Large Language Models (LLMs) to generate future motion from agents' past/observed trajectories and scene semantics. LLMs' powerful comprehension abilities capture a spectrum of high-level scene knowledge and interactive information. Emulating the human-like lane focus cognitive function, we introduce lane-aware probabilistic learning powered by the pioneering Mamba module.
  arXiv  Detail & Related papers  (2024-05-08T09:28:04Z)
• Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization [165.98557106089777]
  A key challenge is to enhance the capabilities of large language models (LLMs) amid a looming shortage of high-quality training data. Our study starts from an empirical strategy for the light continual training of LLMs using their original pre-training data sets. We then formalize this strategy into a principled framework of Instance-Reweighted Distributionally Robust Optimization.
  arXiv  Detail & Related papers  (2024-02-22T04:10:57Z)
• Large Language Models Are Semi-Parametric Reinforcement Learning Agents [15.908831573619842]
  REMEMBERER is capable of exploiting the experiences from the past episodes even for different task goals. Reinforcement Learning with Experience Memory (RLEM) is introduced to update the memory. Experiments are conducted on two RL task sets to evaluate the proposed framework.
  arXiv  Detail & Related papers  (2023-06-09T08:08:18Z)
• ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning [73.47165576175541]
  Two-Tower Vision-Language (VL) models have shown promising improvements on various downstream tasks. We propose ManagerTower, a novel VL model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels.
  arXiv  Detail & Related papers  (2023-05-31T18:23:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.