Metamorphic Testing of Vision-Language Action-Enabled Robots

• URL: http://arxiv.org/abs/2602.22579v1
• Date: Thu, 26 Feb 2026 03:32:43 GMT
• Title: Metamorphic Testing of Vision-Language Action-Enabled Robots
• Abstract summary: We propose Metamorphic Testing (MT) to alleviate the test oracle problem in Vision-Language-Action (VLA) models.<n>An empirical study involving five VLA models, two simulated robots, and four robotic tasks shows that MT can effectively alleviate the test oracle problem.
• Score: 12.232630336816463
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models are multimodal robotic task controllers that, given an instruction and visual inputs, produce a sequence of low-level control actions (or motor commands) enabling a robot to execute the requested task in the physical environment. These systems face the test oracle problem from multiple perspectives. On the one hand, a test oracle must be defined for each instruction prompt, which is a complex and non-generalizable approach. On the other hand, current state-of-the-art oracles typically capture symbolic representations of the world (e.g., robot and object states), enabling the correctness evaluation of a task, but fail to assess other critical aspects, such as the quality with which VLA-enabled robots perform a task. In this paper, we explore whether Metamorphic Testing (MT) can alleviate the test oracle problem in this context. To do so, we propose two metamorphic relation patterns and five metamorphic relations to assess whether changes to the test inputs impact the original trajectory of the VLA-enabled robots. An empirical study involving five VLA models, two simulated robots, and four robotic tasks shows that MT can effectively alleviate the test oracle problem by automatically detecting diverse types of failures, including, but not limited to, uncompleted tasks. More importantly, the proposed MRs are generalizable, making the proposed approach applicable across different VLA models, robots, and tasks, even in the absence of test oracles.

Related papers

• MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models [9.988404944720715]
  Existing testing approaches for VLA-enabled robots rely on manually constructed symbolic test oracles.<n>We introduce MANGO, a multi-agent framework that automatically generates fine-grained oracles from natural-language descriptions.<n>We evaluate MANGO on the LIBERO_10 and RoboCasa Humanoid benchmarks.
  arXiv  Detail & Related papers  (2026-06-23T17:00:06Z)
• Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations [76.79742393097358]
  Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics.<n>Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics.<n>Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task.
  arXiv  Detail & Related papers  (2025-11-27T18:50:21Z)
• ERR@HRI 2.0 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Conversations [18.151307410451796]
  ERR@HRI 2.0 Challenge provides a dataset of conversational robot failures during human-robot conversations.<n> dataset includes 16 hours of dyadic human-robot interactions, incorporating facial, speech, and head movement features.<n>Participants are invited to form teams and develop machine learning models that detect these failures using multimodal data.
  arXiv  Detail & Related papers  (2025-07-17T18:21:45Z)
• ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model [21.844214660424175]
  ChatVLA is a novel framework featuring Phased Alignment Training, which incrementally integrates multimodal data after initial control mastery, and a Mixture-of-Experts architecture to minimize task interference.<n>ChatVLA demonstrates competitive performance on visual question-answering datasets and significantly surpasses state-of-the-art vision-language-action (VLA) methods on multimodal understanding benchmarks.<n>Our findings highlight the potential of our unified framework for achieving both robust multimodal understanding and effective robot control.
  arXiv  Detail & Related papers  (2025-02-20T10:16:18Z)
• Embodied Red Teaming for Auditing Robotic Foundation Models [39.23336328126245]
  Embodied Red Teaming (ERT) generates diverse and challenging instructions to test language-conditioned robot models.<n>ERT uses automated red teaming techniques with Vision Language Models (VLMs) to create contextually grounded, difficult instructions.<n> Experimental results show that state-of-the-art language-conditioned robot models fail or behave unsafely on ERT-generated instructions.
  arXiv  Detail & Related papers  (2024-11-27T18:57:26Z)
• Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models [81.55156507635286]
  Legged robots are physically capable of navigating a diverse variety of environments and overcoming a wide range of obstructions. Current learning methods often struggle with generalization to the long tail of unexpected situations without heavy human supervision. We propose a system, VLM-Predictive Control (VLM-PC), combining two key components that we find to be crucial for eliciting on-the-fly, adaptive behavior selection.
  arXiv  Detail & Related papers  (2024-07-02T21:00:30Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations.<n>First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets.<n>We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)
• On the Vulnerability of LLM/VLM-Controlled Robotics [54.57914943017522]
  We highlight vulnerabilities in robotic systems integrating large language models (LLMs) and vision-language models (VLMs) due to input modality sensitivities.<n>Our results show that simple input perturbations reduce task execution success rates by 22.2% and 14.6% in two representative LLM/VLM-controlled robotic systems.
  arXiv  Detail & Related papers  (2024-02-15T22:01:45Z)
• QUAR-VLA: Vision-Language-Action Model for Quadruped Robots [37.952398683031895]
  The central idea is to elevate the overall intelligence of the robot.<n>We propose QUAdruped Robotic Transformer (QUART), a family of VLA models to integrate visual information and instructions from diverse modalities as input.<n>Our approach leads to performant robotic policies and enables QUART to obtain a range of emergent capabilities.
  arXiv  Detail & Related papers  (2023-12-22T06:15:03Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation [50.737355245505334]
  We propose a novel framework for learning high-level cognitive capabilities in robot manipulation tasks. The resulting dataset AlphaBlock consists of 35 comprehensive high-level tasks of multi-step text plans and paired observation.
  arXiv  Detail & Related papers  (2023-05-30T09:54:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.