RoboReason: Boosting Robotics-Relevant VLM Reasoning Through Scalable Data Generation
Abstract Overview
This paper studies robotics-relevant visual reasoning in vision-language models using three ordering tasks built from real robot camera data: temporal frame ordering, depth ordering, and left-to-right object ordering. The tasks use stereo and wrist-camera imagery from robot demonstrations and vary difficulty by increasing the number of images or objects to sort. The authors first show that several strong VLMs perform poorly on these real-world spatiotemporal ordering problems, especially as set size grows. They then introduce RoboReason, a scalable pipeline that automatically generates training data from robot trajectories and use it to finetune smaller VLMs with supervised finetuning and reinforcement learning. The resulting models improve on the ordering tasks and show some gains on downstream visual and robotics reasoning benchmarks, though improvements are not uniform across all tasks.
Novelty
The distinctive contribution is a scalable data-generation pipeline that turns robot manipulation trajectories into spatiotemporal ordering tasks aligned with robot-centric visual reasoning. The work also frames ordering from multi-view robot cameras as a practical evaluation and training substrate for improving VLM reasoning relevant to robotics.
Results
Across zero-shot evaluations, state-of-the-art VLMs struggled on the proposed tasks, with accuracy dropping sharply as more items had to be ordered; even 2-item settings often stayed below 80% for some leading models. Adding segmentation or bounding-box overlays generally helped spatial ordering more than temporal ordering, and merge-sort-style structured test-time compute mainly benefited spatial tasks. Finetuning smaller models on RoboReason-generated data substantially improved spatial ordering, sometimes matching or surpassing much larger VLMs on 3-object spatial tasks, while downstream gains were mixed and temporal-ordering improvements were more limited.
Key Points
- Real-world robot-camera ordering tasks expose substantial weaknesses in current VLM spatiotemporal reasoning, particularly as ordering set cardinality increases.
- RoboReason automatically constructs temporal, depth, and left-to-right ordering datasets from DROID trajectories using object detection, segmentation, and depth estimation tools.
- Training on generated data via SFT or GRPO notably boosts small-model spatial reasoning and yields some downstream benefits, but improvements are task-dependent and the pipeline can introduce labeling or ordering errors.