A Taxonomy for Evaluating Generalist Robot Policies

• URL: http://arxiv.org/abs/2503.01238v1
• Date: Mon, 03 Mar 2025 07:03:00 GMT
• Title: A Taxonomy for Evaluating Generalist Robot Policies
• Abstract summary: evaluating and quantifying generalization is the Wild West of modern robotics.<n>We propose STAR-Gen, a taxonomy of generalization for robot manipulation structured around visual, semantic, and behavioral generalizations.<n>We evaluate a variety of state-of-the-art models on a benchmark to demonstrate the utility of our taxonomy in practice.
• Score: 29.437786401917702
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine learning for robotics promises to unlock generalization to novel tasks and environments. Guided by this promise, many recent works have focused on scaling up robot data collection and developing larger, more expressive policies to achieve this. But how do we measure progress towards this goal of policy generalization in practice? Evaluating and quantifying generalization is the Wild West of modern robotics, with each work proposing and measuring different types of generalization in their own, often difficult to reproduce, settings. In this work, our goal is (1) to outline the forms of generalization we believe are important in robot manipulation in a comprehensive and fine-grained manner, and (2) to provide reproducible guidelines for measuring these notions of generalization. We first propose STAR-Gen, a taxonomy of generalization for robot manipulation structured around visual, semantic, and behavioral generalization. We discuss how our taxonomy encompasses most prior notions of generalization in robotics. Next, we instantiate STAR-Gen with a concrete real-world benchmark based on the widely-used Bridge V2 dataset. We evaluate a variety of state-of-the-art models on this benchmark to demonstrate the utility of our taxonomy in practice. Our taxonomy of generalization can yield many interesting insights into existing models: for example, we observe that current vision-language-action models struggle with various types of semantic generalization, despite the promise of pre-training on internet-scale language datasets. We believe STAR-Gen and our guidelines can improve the dissemination and evaluation of progress towards generalization in robotics, which we hope will guide model design and future data collection efforts. We provide videos and demos at our website stargen-taxonomy.github.io.

Related papers

• The Embodiment Gap in Robot Foundation Models [7.90590547085857]
  In robotics, a model can generalize while work still remains before it can run on a robot with a particular body.<n>We call the gap between reusable models, representations, or data and their use in execution on the target robot the embodiment gap.<n>This survey examines what can be reused across robot embodiments and what must still be implemented on a new robot.
  arXiv  Detail & Related papers  (2026-08-19T01:55:04Z)
• Inductive Generalization for Robotic Manipulation [27.207954193702008]
  We argue that to measure generalization capabilities we must instead test the inductive capacity of policies on progressively harder, out-of-distribution task variants.<n>We provide a reusable and formal evaluation protocol for measuring inductive generalization in any manipulation policy.
  arXiv  Detail & Related papers  (2026-06-19T00:19:53Z)
• World Model for Robot Learning: A Comprehensive Survey [168.5691568303064]
  This survey systematically reviews the rapidly growing literature on world models for robot learning.<n>We examine how world models are coupled with robot policies, how they serve as learned simulators for reinforcement learning and evaluation, and how robotic video world models have progressed from imagination-based generation to controllable, structured, and foundation-scale formulations.<n>Overall, this survey systematically reviews the rapidly growing literature on world models for robot learning, clarifies key paradigms and applications, and highlights major challenges and future directions for predictive modeling in embodied agents.
  arXiv  Detail & Related papers  (2026-04-30T14:35:31Z)
• Generalizable Geometric Prior and Recurrent Spiking Feature Learning for Humanoid Robot Manipulation [90.90219129619344]
  This paper presents a novel R-prior-S, Recurrent Geometric-priormodal Policy with Spiking features.<n>To ground high-level reasoning in physical reality, we leverage lightweight 2D geometric inductive biases.<n>For the data efficiency issue in robotic action generation, we introduce a Recursive Adaptive Spiking Network.
  arXiv  Detail & Related papers  (2026-01-13T23:36:30Z)
• Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations [26.678553477485362]
  We present a framework that better preserves pretrained features while adapting them for robot manipulation.<n>Our approach introduces three components: (i) a dual-encoder design with one frozen vision to retain pretrained features and another trainable for task adaptation, (ii) a string-based action tokenizer that casts continuous actions into character sequences aligned with the model's pretraining domain, and (iii) a co-training strategy that combines robot demonstrations with vision-language datasets emphasizing spatial reasoning and affordances.
  arXiv  Detail & Related papers  (2025-09-14T20:08:56Z)
• From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models [5.660635614478238]
  Vision-Language-Action (VLA) models promise to produce versatile, "generalist" robot policies.<n>Traditional imitation learning benchmarks are unsuitable due to the lack of language instructions.<n>We introduce a unified suite of 50 simulation-based tasks across 10 subcategories spanning language instruction, vision, and objects.
  arXiv  Detail & Related papers  (2025-06-11T16:52:18Z)
• $π_{0.5}$: a Vision-Language-Action Model with Open-World Generalization [81.73746512639283]
  We describe a new model based on $pi_0.5$ that uses co-training on heterogeneous tasks to enable broad generalization. We demonstrate for the first time that an end-to-end learning-enabled robotic system can perform long-horizon and dexterous manipulation skills.
  arXiv  Detail & Related papers  (2025-04-22T17:31:29Z)
• DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping [14.511049253735834]
  A general-purpose robot must be capable of grasping diverse objects in arbitrary scenarios. Our solution is DexGraspVLA, a hierarchical framework that utilizes a pre-trained Vision-Language model as the high-level task planner. Our method achieves a 90+% success rate under thousands of unseen object, lighting, and background combinations.
  arXiv  Detail & Related papers  (2025-02-28T09:57:20Z)
• Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding [85.63710017456792]
  FuSe is a novel approach that enables finetuning visuomotor generalist policies on heterogeneous sensor modalities. We show that FuSe enables performing challenging tasks that require reasoning jointly over modalities such as vision, touch, and sound. Experiments in the real world show that FuSeis able to increase success rates by over 20% compared to all considered baselines.
  arXiv  Detail & Related papers  (2025-01-08T18:57:33Z)
• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• Grounding Robot Policies with Visuomotor Language Guidance [15.774237279917594]
  We propose an agent-based framework for grounding robot policies to the current context. The proposed framework is composed of a set of conversational agents designed for specific roles. We demonstrate that our approach can effectively guide manipulation policies to achieve significantly higher success rates.
  arXiv  Detail & Related papers  (2024-10-09T02:00:37Z)
• Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis [82.59451639072073]
  General-purpose robots operate seamlessly in any environment, with any object, and utilize various skills to complete diverse tasks. As a community, we have been constraining most robotic systems by designing them for specific tasks, training them on specific datasets, and deploying them within specific environments. Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models, we devote this survey to exploring how foundation models can be applied to general-purpose robotics.
  arXiv  Detail & Related papers  (2023-12-14T10:02:55Z)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [140.48218261864153]
  We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control. Our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training.
  arXiv  Detail & Related papers  (2023-07-28T21:18:02Z)
• Transferring Foundation Models for Generalizable Robotic Manipulation [82.12754319808197]
  We propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models.<n>Our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning.<n>Demos can be found in our submitted video, and more comprehensive ones can be found in link1 or link2.
  arXiv  Detail & Related papers  (2023-06-09T07:22:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.