Contrast Sets for Evaluating Language-Guided Robot Policies

• URL: http://arxiv.org/abs/2406.13636v2
• Date: Fri, 25 Oct 2024 15:23:07 GMT
• Title: Contrast Sets for Evaluating Language-Guided Robot Policies
• Abstract summary: We introduce contrast sets for robotics to make small, but specific, perturbations to otherwise independent, identically distributed test instances. We investigate the relationship between experimenter effort to carry out an evaluation and the resulting estimated test performance. We use the relative performance change of different contrast set perturbations to characterize policies at reduced experimenter effort in both a simulated manipulation task and a physical robot vision-and-language navigation task.
• Score: 10.105811371008091
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robot evaluations in language-guided, real world settings are time-consuming and often sample only a small space of potential instructions across complex scenes. In this work, we introduce contrast sets for robotics as an approach to make small, but specific, perturbations to otherwise independent, identically distributed (i.i.d.) test instances. We investigate the relationship between experimenter effort to carry out an evaluation and the resulting estimated test performance as well as the insights that can be drawn from performance on perturbed instances. We use the relative performance change of different contrast set perturbations to characterize policies at reduced experimenter effort in both a simulated manipulation task and a physical robot vision-and-language navigation task. We encourage the use of contrast set evaluations as a more informative alternative to small scale, i.i.d. demonstrations on physical robots, and as a scalable alternative to industry-scale real world evaluations.

Related papers

• AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization [88.25615187835321]
  We propose AnyWorld, a cross-embodiment world modeling framework.<n>Our model factorizes an interaction into action, camera, and embodiment.<n>We train the model with large-scale human interaction pretraining.
  arXiv  Detail & Related papers  (2026-08-29T12:55:15Z)
• IMBench: A Benchmark for Intuitive Robotic Manipulation [11.843368723732679]
  Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints.<n>Existing benchmarks fail to capture this integration.<n>We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution.
  arXiv  Detail & Related papers  (2026-07-17T05:34:29Z)
• When Absolute State Fails: Evaluating Proprioceptive Encodings for Robust Manipulation [0.14990005092937678]
  We study strategies for encoding the robot's proprioceptive state to improve both in- and out-of-distribution performance at test time.<n>We find that a simple episode-wise relative frame provides the best trade-off between task performance and robustness.
  arXiv  Detail & Related papers  (2026-05-13T06:41:56Z)
• Robotic Scene Cloning:Advancing Zero-Shot Robotic Scene Adaptation in Manipulation via Visual Prompt Editing [76.13853194607252]
  We propose Robotic Scene Cloning (RSC), a novel method designed for scene-specific adaptation by editing existing robot operation trajectories.<n>RSC achieves accurate and scene-consistent sample generation by leveraging a visual prompting mechanism and a carefully tuned condition injection module.<n> Experiments across various simulated and real-world environments demonstrate that RSC significantly enhances policy generalization in target environments.
  arXiv  Detail & Related papers  (2026-03-10T14:21:25Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• RobotArena $\infty$: Scalable Robot Benchmarking via Real-to-Sim Translation [47.79800816696372]
  Real-world testing of manipulation policies is labor-intensive at scale, and difficult to reproduce.<n>Existing simulation benchmarks are similarly limited, as they train and test policies within the same synthetic domains.<n>In this paper, we introduce a new benchmarking framework that overcomes these challenges by shifting VLA evaluation into large-scale simulated augmented environments.
  arXiv  Detail & Related papers  (2025-10-27T17:41:38Z)
• Recognizing Actions from Robotic View for Natural Human-Robot Interaction [52.00935005918032]
  Natural Human-Robot Interaction (N-HRI) requires robots to recognize human actions at varying distances and states, regardless of whether the robot itself is in motion or stationary.<n>Existing benchmarks for N-HRI fail to address the unique complexities in N-HRI due to limited data, modalities, task categories, and diversity of subjects and environments.<n>We introduce (Action from Robotic View) a large-scale dataset for perception-centric robotic views prevalent in mobile service robots.
  arXiv  Detail & Related papers  (2025-07-30T09:48:34Z)
• Learning to Evaluate Autonomous Behaviour in Human-Robot Interaction [26.765285050287247]
  We propose a general evaluation framework that measures the quality of Imitation Learning (IL) methods by focusing on trajectory performance.<n>We devise the Neural Meta Evaluator (NeME), a deep learning model trained to classify actions from robot joint trajectories.<n>We validate our framework on ergoCub, a humanoid robot, using teleoperation data and comparing IL methods tailored to the available platform.
  arXiv  Detail & Related papers  (2025-07-08T21:12:57Z)
• Is an object-centric representation beneficial for robotic manipulation ? [45.75998994869714]
  Object-centric representation (OCR) has recently become a subject of interest in the computer vision community for learning a structured representation of images and videos.<n>We evaluate one classical object-centric method across several generalization scenarios and compare its results against several state-of-the-art hollistic representations.<n>Our results exhibit that existing methods are prone to failure in difficult scenarios involving complex scene structures, whereas object-centric methods help overcome these challenges.
  arXiv  Detail & Related papers  (2025-06-24T08:23:55Z)
• Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selection [26.01454902727706]
  evaluating learned robot control policies to determine their physical task-level capabilities costs experimenter time and effort. It is impractical to test every policy on every task multiple times; each trial requires a manual environment reset. We propose to model the distribution of robot performance across all tasks and policies as we sequentially execute experiments.
  arXiv  Detail & Related papers  (2025-02-14T00:07:02Z)
• DRAPER: Towards a Robust Robot Deployment and Reliable Evaluation for Quasi-Static Pick-and-Place Cloth-Shaping Neural Controllers [2.720296126199296]
  This study demonstrates a reliable real-world comparison of different simulation-trained neural controllers on flattening and folding tasks. We introduce the DRAPER framework to enable this comprehensive study, which reliably reflects the true capabilities of these neural controllers.
  arXiv  Detail & Related papers  (2024-09-23T16:08:16Z)
• DiffGen: Robot Demonstration Generation via Differentiable Physics Simulation, Differentiable Rendering, and Vision-Language Model [72.66465487508556]
  DiffGen is a novel framework that integrates differentiable physics simulation, differentiable rendering, and a vision-language model. It can generate realistic robot demonstrations by minimizing the distance between the embedding of the language instruction and the embedding of the simulated observation. Experiments demonstrate that with DiffGen, we could efficiently and effectively generate robot data with minimal human effort or training time.
  arXiv  Detail & Related papers  (2024-05-12T15:38:17Z)
• Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks [0.0]
  In this work, we focus on unsupervised vision-language--action mapping in the area of robotic manipulation. We propose a model-invariant training alternative that improves the models' performance in a simulator by up to 55%. Our work thus also sheds light on the potential benefits and limitations of using the current multimodal VAEs for unsupervised learning of robotic motion trajectories.
  arXiv  Detail & Related papers  (2024-04-02T13:25:16Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• What Matters to You? Towards Visual Representation Alignment for Robot Learning [81.30964736676103]
  When operating in service of people, robots need to optimize rewards aligned with end-user preferences. We propose Representation-Aligned Preference-based Learning (RAPL), a method for solving the visual representation alignment problem.
  arXiv  Detail & Related papers  (2023-10-11T23:04:07Z)
• SCENEREPLICA: Benchmarking Real-World Robot Manipulation by Creating Replicable Scenes [5.80109297939618]
  We present a new reproducible benchmark for evaluating robot manipulation in the real world, specifically focusing on pick-and-place. Our benchmark uses the YCB objects, a commonly used dataset in the robotics community, to ensure that our results are comparable to other studies.
  arXiv  Detail & Related papers  (2023-06-27T16:59:15Z)
• Inferring Versatile Behavior from Demonstrations by Matching Geometric Descriptors [72.62423312645953]
  Humans intuitively solve tasks in versatile ways, varying their behavior in terms of trajectory-based planning and for individual steps. Current Imitation Learning algorithms often only consider unimodal expert demonstrations and act in a state-action-based setting. Instead, we combine a mixture of movement primitives with a distribution matching objective to learn versatile behaviors that match the expert's behavior and versatility.
  arXiv  Detail & Related papers  (2022-10-17T16:42:59Z)
• An in-depth experimental study of sensor usage and visual reasoning of robots navigating in real environments [20.105395754497202]
  We study the performance and reasoning capacities of real physical agents, trained in simulation and deployed to two different physical environments. We show, that for the PointGoal task, an agent pre-trained on wide variety of tasks and fine-tuned on a simulated version of the target environment can reach competitive performance without modelling any sim2real transfer.
  arXiv  Detail & Related papers  (2021-11-29T16:27:29Z)
• Improving Object Permanence using Agent Actions and Reasoning [8.847502932609737]
  Existing approaches learn object permanence from low-level perception. We argue that object permanence can be improved when the robot uses knowledge about executed actions.
  arXiv  Detail & Related papers  (2021-10-01T07:09:49Z)
• Few-Shot Visual Grounding for Natural Human-Robot Interaction [0.0]
  We propose a software architecture that segments a target object from a crowded scene, indicated verbally by a human user. At the core of our system, we employ a multi-modal deep neural network for visual grounding. We evaluate the performance of the proposed model on real RGB-D data collected from public scene datasets.
  arXiv  Detail & Related papers  (2021-03-17T15:24:02Z)
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation [115.4071729927011]
  We study the effects of using mid-level visual representations as generic and easy-to-decode perceptual state in an end-to-end RL framework. We show that they aid generalization, improve sample complexity, and lead to a higher final performance. In practice, this means that mid-level representations could be used to successfully train policies for tasks where domain randomization and learning-from-scratch failed.
  arXiv  Detail & Related papers  (2020-11-13T00:16:05Z)
• Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents [61.36681529571202]
  We describe a new concept for reproducible robotics research that integrates development and benchmarking. One of the central components of this setup is the Duckietown Autolab, a standardized setup that is itself relatively low-cost and reproducible. We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.
  arXiv  Detail & Related papers  (2020-09-09T15:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.