RoboVista: Evaluating Vision Language Models for Diverse Robot Applications

• URL: http://arxiv.org/abs/2607.04610v1
• Date: Mon, 06 Jul 2026 02:34:56 GMT
• Title: RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
• Abstract summary: Vision-Language Models (VLMs) offer a promising foundation for general-purpose and interpretable robotic reasoning.<n>We propose Robot Question Answering (RQA) and RoboVista, a benchmark curated from real robotic systems, research papers, and expert annotations.
• Score: 27.371058136378718
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Diverse applications for robotics, such as industry and agriculture, require robots to operate across various embodiments, changing visual conditions, and complex planning. Vision-Language Models (VLMs) offer a promising foundation for general-purpose and interpretable robotic reasoning. Aligning VLMs with diverse robot applications requires a modular understanding of the individual decision components that underlie robotic behavior. Capturing such structure is challenging for conventional robot benchmarks that are primarily based on teleoperated, end-to-end datasets. We propose Robot Question Answering (RQA), a modular evaluation framework and RoboVista, a benchmark curated from real robotic systems, research papers, and expert annotations. RoboVista contains 474 Visual Question Answering (VQA) instances with human annotated reasoning and covers 39 unique task types in agricultural, industrial, domestic, surgical robotics, autonomous driving, and open robot datasets. Experiments on RoboVista show that state-of-the-art VLMs exhibit substantial gaps. Physical robot experiments suggest strong correlation between RoboVista performance and real-world task execution.

Related papers

• Robots Need More than VLA and World Models [38.16463528269755]
  Generalist robot intelligence is often framed as a policy-scaling problem.<n>In this paper we argue that this framing is incomplete.<n>We identify four missing components for the next generation of robotics.
  arXiv  Detail & Related papers  (2026-06-04T10:43:14Z)
• RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video [56.9581053843815]
  We introduce RobotSeg, a foundation model for robot segmentation in image and video.<n>It addresses the lack of adaptation to articulated robots, reliance on manual prompts, and the need for per-frame training mask annotations.<n>It achieves state-of-the-art performance on both images and videos.
  arXiv  Detail & Related papers  (2025-11-28T07:51:02Z)
• Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations [76.79742393097358]
  Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics.<n>Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics.<n>Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task.
  arXiv  Detail & Related papers  (2025-11-27T18:50:21Z)
• TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models [1.534667887016089]
  The presented paper investigates recent advancements in Large Language Models (LLMs) and Vision Language Models (VLMs)<n>This integration allows robots to understand and execute commands given in natural language and to perceive their environment through visual and/or descriptive inputs.<n>Our paper outlines four LLM-assisted simulated robotic control, which explore (i) low-level control, (ii) the generation of language-based feedback that describes the robot's internal states, (iii) the use of visual information as additional input, and (iv) the use of robot structure information for generating task plans and feedback.
  arXiv  Detail & Related papers  (2024-12-19T23:43:40Z)
• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models [81.55156507635286]
  Legged robots are physically capable of navigating a diverse variety of environments and overcoming a wide range of obstructions. Current learning methods often struggle with generalization to the long tail of unexpected situations without heavy human supervision. We propose a system, VLM-Predictive Control (VLM-PC), combining two key components that we find to be crucial for eliciting on-the-fly, adaptive behavior selection.
  arXiv  Detail & Related papers  (2024-07-02T21:00:30Z)
• RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation [77.41969287400977]
  This paper presents textbfRobotScript, a platform for a deployable robot manipulation pipeline powered by code generation. We also present a benchmark for a code generation benchmark for robot manipulation tasks in free-form natural language. We demonstrate the adaptability of our code generation framework across multiple robot embodiments, including the Franka and UR5 robot arms.
  arXiv  Detail & Related papers  (2024-02-22T15:12:00Z)
• QUAR-VLA: Vision-Language-Action Model for Quadruped Robots [37.952398683031895]
  The central idea is to elevate the overall intelligence of the robot.<n>We propose QUAdruped Robotic Transformer (QUART), a family of VLA models to integrate visual information and instructions from diverse modalities as input.<n>Our approach leads to performant robotic policies and enables QUART to obtain a range of emergent capabilities.
  arXiv  Detail & Related papers  (2023-12-22T06:15:03Z)
• WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model [92.90127398282209]
  This paper investigates the potential of integrating the most recent Large Language Models (LLMs) and existing visual grounding and robotic grasping system. We introduce the WALL-E (Embodied Robotic WAiter load lifting with Large Language model) as an example of this integration. We deploy this LLM-empowered system on the physical robot to provide a more user-friendly interface for the instruction-guided grasping task.
  arXiv  Detail & Related papers  (2023-08-30T11:35:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.