Grounding Robot Generalization in Training Data via Retrieval-Augmented VLMs

• URL: http://arxiv.org/abs/2603.11426v2
• Date: Wed, 18 Mar 2026 17:58:07 GMT
• Title: Grounding Robot Generalization in Training Data via Retrieval-Augmented VLMs
• Abstract summary: RADAR is a framework for comparing test-time evaluation tasks to policy training data.<n>We show that vision-language models (VLMs) are effective at analyzing data for generalization.<n>We scale RADAR to large-scale datasets, where we observe agreement with human-defined benchmark conditions.
• Score: 28.877758891217184
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work on robot manipulation has advanced policy generalization to novel scenarios. However, it is often difficult to characterize how different evaluation settings actually represent generalization from the training distribution of a given policy. To work towards more precise evaluation of generalization in robotics, we propose RADAR, a scalable framework for directly comparing test-time evaluation tasks to policy training data, to determine what form of policy generalization is required. RADAR consists of a two-stage pipeline: first, retrieval using generalist policy embeddings identifies which training examples are relevant for a given evaluation task. Next, vision-language models (VLMs) analyze the evaluation task against the retrieved data, outputting interpretable analysis on how they compare along a variety of axes, and an overall classification of what type of policy generalization is required. Through controlled experiments, we demonstrate that VLMs are effective at analyzing data for generalization, and that our retrieval step effectively identifies examples needed to make accurate classifications with respect to the training data. Furthermore, we scale RADAR to large-scale datasets, where we observe agreement with human-defined benchmark conditions from prior work. We provide demonstrations at radar-analysis.github.io.

Related papers

• Inductive Generalization for Robotic Manipulation [27.207954193702008]
  We argue that to measure generalization capabilities we must instead test the inductive capacity of policies on progressively harder, out-of-distribution task variants.<n>We provide a reusable and formal evaluation protocol for measuring inductive generalization in any manipulation policy.
  arXiv  Detail & Related papers  (2026-06-19T00:19:53Z)
• Can Generalist Agents Automate Data Curation? [58.652262227632406]
  Curating training data is among the most consequential yet labor-intensive parts of modern AI development.<n>We ask whether generalist coding agents can automate this data-curation loop.<n>We introduce *Curation-Bench*, an agent-centric benchmark that fixes the model, training recipe, and evaluation suite.
  arXiv  Detail & Related papers  (2026-06-02T22:26:53Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation [117.54237701533805]
  Generalist robot policies trained on large-scale datasets such as Open X-Embodiment (OXE) demonstrate strong performance across a wide range of tasks.<n>They often struggle to generalize beyond the distribution of their training data.<n>We identify shortcut learning as a key impediment to generalization.
  arXiv  Detail & Related papers  (2025-08-08T16:14:01Z)
• RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies [125.35572632340602]
  We propose RoboArena, a new approach for scalable evaluation of generalist robot policies in the real world.<n>Instead of standardizing evaluations around fixed tasks, environments, or locations, we propose to crowd-source evaluations across a distributed network of evaluators.<n>We instantiate our approach across a network of evaluators at seven academic institutions using the DROID robot platform.
  arXiv  Detail & Related papers  (2025-06-22T18:13:31Z)
• STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learning [8.860366821983211]
  STRAP is a technique for leveraging pre-trained vision foundation models and dynamic time warping to retrieve sub-sequences of trajectories from large training corpora in a robust fashion.<n>This work proposes STRAP, a technique for leveraging pre-trained vision foundation models and dynamic time warping to retrieve sub-sequences of trajectories from large training corpora in a robust fashion.
  arXiv  Detail & Related papers  (2024-12-19T18:54:06Z)
• Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance [66.51390591688802]
  Value-Guided Policy Steering (V-GPS) is compatible with a wide range of different generalist policies, without needing to fine-tune or even access the weights of the policy.<n>We show that the same value function can improve the performance of five different state-of-the-art policies with different architectures.
  arXiv  Detail & Related papers  (2024-10-17T17:46:26Z)
• DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors [13.700885996266457]
  Learning from previously collected data via behavioral cloning or offline reinforcement learning (RL) is a powerful recipe for scaling generalist agents. We present theDeepMind Control Visual Benchmark (DMC-VB), a dataset collected in the DeepMind Control Suite to evaluate the robustness of offline RL agents. Accompanying our dataset, we propose three benchmarks to evaluate representation learning methods for pretraining, and carry out experiments on several recently proposed methods.
  arXiv  Detail & Related papers  (2024-09-26T23:07:01Z)
• Foundation Policies with Hilbert Representations [54.44869979017766]
  We propose an unsupervised framework to pre-train generalist policies from unlabeled offline data. Our key insight is to learn a structured representation that preserves the temporal structure of the underlying environment. Our experiments show that our unsupervised policies can solve goal-conditioned and general RL tasks in a zero-shot fashion.
  arXiv  Detail & Related papers  (2024-02-23T19:09:10Z)
• BIRB: A Generalization Benchmark for Information Retrieval in Bioacoustics [7.68184437595058]
  We present BIRB, a complex benchmark centered on the retrieval of bird vocalizations from passively-recorded datasets. We propose a baseline system for this collection of tasks using representation learning and a nearest-centroid search.
  arXiv  Detail & Related papers  (2023-12-12T17:06:39Z)
• DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks [88.62288327934499]
  We propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-11-03T07:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.