Benchmarking Affordance Generalization with BusyBox

• URL: http://arxiv.org/abs/2602.05441v1
• Date: Thu, 05 Feb 2026 08:31:27 GMT
• Title: Benchmarking Affordance Generalization with BusyBox
• Abstract summary: Vision-Language-Action (VLA) models have been attracting the attention of researchers and practitioners thanks to their promise of generalization.<n>We present BusyBox, a physical benchmark for systematic evaluation of VLAs' affordance generalization.<n>We empirically demonstrate that generalization across BusyBox variants is highly challenging even for strong open-weights.
• Score: 12.034349600313556
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models have been attracting the attention of researchers and practitioners thanks to their promise of generalization. Although single-task policies still offer competitive performance, VLAs are increasingly able to handle commands and environments unseen in their training set. While generalization in vision and language space is undoubtedly important for robust versatile behaviors, a key meta-skill VLAs need to possess is affordance generalization -- the ability to manipulate new objects with familiar physical features. In this work, we present BusyBox, a physical benchmark for systematic semi-automatic evaluation of VLAs' affordance generalization. BusyBox consists of 6 modules with switches, sliders, wires, buttons, a display, and a dial. The modules can be swapped and rotated to create a multitude of BusyBox variations with different visual appearances but the same set of affordances. We empirically demonstrate that generalization across BusyBox variants is highly challenging even for strong open-weights VLAs such as $π_{0.5}$ and GR00T-N1.6. To encourage the research community to evaluate their own VLAs on BusyBox and to propose new affordance generalization experiments, we have designed BusyBox to be easy to build in most robotics labs. We release the full set of CAD files for 3D-printing its parts as well as a bill of materials for (optionally) assembling its electronics. We also publish a dataset of language-annotated demonstrations that we collected using the common bimanual Mobile Aloha robot on the canonical BusyBox configuration. All of the released materials are available at https://microsoft.github.io/BusyBox.

Related papers

• Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Models [7.1750939299528795]
  Vision-Language-Action (VLA) models have emerged as a promising approach for general-purpose robot manipulation.<n>We apply mechanistic interpretability techniques to better understand the inner workings of VLA models.<n>We show that individual SAE features causally influence robot behavior.
  arXiv  Detail & Related papers  (2026-03-19T17:42:05Z)
• Maestro: Orchestrating Robotics Modules with Vision-Language Models for Zero-Shot Generalist Robots [54.62646284378409]
  We build policies around vision-language models (VLMs) by augmenting their general capabilities with specific robot capabilities encapsulated in a curated set of perception, planning, and control modules.<n>In Maestro, a VLM coding agent dynamically composes these modules into a programmatic policy for the current task and scenario.
  arXiv  Detail & Related papers  (2025-11-02T12:34:37Z)
• From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models [5.660635614478238]
  Vision-Language-Action (VLA) models promise to produce versatile, "generalist" robot policies.<n>Traditional imitation learning benchmarks are unsuitable due to the lack of language instructions.<n>We introduce a unified suite of 50 simulation-based tasks across 10 subcategories spanning language instruction, vision, and objects.
  arXiv  Detail & Related papers  (2025-06-11T16:52:18Z)
• OG-VLA: 3D-Aware Vision Language Action Model via Orthographic Image Generation [68.11862866566817]
  3D-aware policies achieve state-of-the-art performance on precise robot manipulation tasks, but struggle with generalization to unseen instructions, scenes, and objects.<n>We introduce OG-VLA, a novel architecture and learning framework that combines the generalization strengths of Vision Language Action models (VLAs) with the robustness of 3D-aware policies.
  arXiv  Detail & Related papers  (2025-06-01T22:15:45Z)
• $π_{0.5}$: a Vision-Language-Action Model with Open-World Generalization [81.73746512639283]
  We describe a new model based on $pi_0.5$ that uses co-training on heterogeneous tasks to enable broad generalization.<n>We demonstrate for the first time that an end-to-end learning-enabled robotic system can perform long-horizon and dexterous manipulation skills.
  arXiv  Detail & Related papers  (2025-04-22T17:31:29Z)
• JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse [30.10039754181912]
  We introduce Act from Visual Language Post-Training, which refines Visual Language Models (VLMs) through visual and linguistic guidance in a self-supervised manner.<n>This enhancement improves the models' capabilities in world knowledge, visual recognition, and spatial grounding in open-world environments.<n>We obtain the first VLA models in Minecraft that can follow human instructions on over 1k different atomic tasks.
  arXiv  Detail & Related papers  (2025-03-20T17:21:58Z)
• PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Model [4.079327215055764]
  Affordance understanding, the task of identifying actionable regions on 3D objects, plays a vital role in allowing robotic systems to engage with and operate within the physical world.<n>Visual Language Models (VLMs) have excelled in high-level reasoning but fall short in grasping the nuanced physical properties required for effective human-robot interaction.<n>We introduce PAVLM, an innovative framework that utilizes the extensive multimodal knowledge embedded in pre-trained language models to enhance 3D affordance understanding of point cloud.
  arXiv  Detail & Related papers  (2024-10-15T12:53:42Z)
• Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning [21.944363082061333]
  We propose textbfManiwhere, a generalizable framework tailored for visual reinforcement learning. Our experiments show that Maniwhere significantly outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-22T17:29:02Z)
• OpenVLA: An Open-Source Vision-Language-Action Model [131.74098076670103]
  We introduce OpenVLA, an open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA shows strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate. We release model checkpoints, fine-tuning notebooks, and our PyTorch with built-in support for training VLAs at scale on Open X-Embodiment datasets.
  arXiv  Detail & Related papers  (2024-06-13T15:46:55Z)
• OpenBox: A Python Toolkit for Generalized Black-box Optimization [50.88355903338381]
  Black-box optimization (BBO) has a broad range of applications, including automatic machine learning, experimental design, and database knob tuning. This paper presents OpenBox, an open-source BBO toolkit with improved usability. It implements user-friendly interfaces and visualization for users to define and manage their tasks.
  arXiv  Detail & Related papers  (2023-04-26T07:13:50Z)
• Training Vision-Language Transformers from Captions [80.00302205584335]
  We introduce a new model Vision-Language from Captions (VLC) built on top of Masked Auto-Encoders. In a head-to-head comparison between ViLT and our model, we find that our approach outperforms ViLT on standard benchmarks.
  arXiv  Detail & Related papers  (2022-05-19T00:19:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.