Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

• URL: http://arxiv.org/abs/2606.19297v1
• Date: Wed, 17 Jun 2026 17:20:46 GMT
• Title: Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models
• Abstract summary: We introduce Act2Answer, a protocol that adapts VLM knowledge benchmarks to VLA evaluation by requiring agents to answer through action.<n>Each question becomes a short tabletop episode where the agent performs a single object-placement action to select among candidate answers, yielding an action-grounded success rate with reduced control confounds.<n>In a large-scale study of 7 VLA models and 9 VLM baselines, we systematically rank models across categories, finding that VLAs show solid performance on simple concepts while exhibiting larger gaps on richer semantic categories relative to their source VLMs.
• Score: 42.38903028184938
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embodied Vision-Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation. Failures on knowledge-sensitive tasks are ambiguous, conflating missing knowledge with poor generalization of low-level control. We introduce Act2Answer, a lightweight protocol that adapts VLM knowledge benchmarks to VLA evaluation by requiring agents to answer through action. Each question becomes a short tabletop episode where the agent performs a single object-placement action to select among candidate answers, yielding an action-grounded success rate with reduced control confounds. We curate a test suite of such environments across diverse commonsense and world-knowledge categories and introduce layerwise intent probing to localize answer-relevant information across the VLM backbone and action head. In a large-scale study of 7 VLA models and 9 VLM baselines, we systematically rank models across categories, finding that VLAs show solid performance on simple concepts while exhibiting larger gaps on richer semantic categories relative to their source VLMs, that VQA co-training is associated with better knowledge retention, and that answer-relevant signals peak in middle VLA layers but attenuate in upper layers. Act2Answer is available at https://tttonyalpha.github.io/act2answer/.

Related papers

• CometVLA: Co-Training on an Embodied Data Pyramid towards Physical Understanding [56.55196056616493]
  Vision-language-action (VLA) models remain brittle in manipulation tasks that require physical commonsense.<n>It remains unclear whether improved VLM physical understanding actually benefits downstream action generation.<n>We present CometVLA, an embodied physical VQA corpus and benchmark strictly aligned with the robot's action data and embodiment.
  arXiv  Detail & Related papers  (2026-08-31T05:55:34Z)
• QuoVLA: Quotient Space for Vision-Language-Action Models [51.02329790939691]
  Vision-Language-Action (VLA) models commonly adapt pretrained VLM latents to robot control by mapping visual observations and language instructions to continuous actions.<n>Our textitQuotient Theory for VLA shows that pretrained VLM latents are not action-insufficient but action-sufficient.<n>We propose QuoVLA, a quotient-space framework for VLA that compresses pretrained VLM latents into action-sufficient representations.
  arXiv  Detail & Related papers  (2026-05-24T06:28:53Z)
• Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control [46.169163284648384]
  We introduce Steerable Policies: VLAs trained on rich synthetic commands at various levels of abstraction, like subtasks, motions, and grounded pixel coordinates.<n>We demonstrate this benefit by controlling our Steerable Policies with both a learned high-level embodied reasoner and an off-the-shelf VLM prompted to reason over command abstractions via in-context learning.
  arXiv  Detail & Related papers  (2026-02-13T18:57:56Z)
• VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models [43.09726338623949]
  Vision-Language-Action (VLA) models integrate pretrained large Vision-Language Models (VLM) into their policy backbone.<n>This paper revisits a fundamental yet seldom systematically studied question: how VLM choice and competence translate to downstream VLA policies performance.<n>We introduce VLM4VLA, a minimal adaptation pipeline that converts general-purpose VLMs into VLA policies using only a small set of new learnable parameters.
  arXiv  Detail & Related papers  (2026-01-06T09:58:24Z)
• How Do VLAs Effectively Inherit from VLMs? [28.72002932514493]
  Vision-language-action (VLA) models hold the promise to attain generalizable embodied control.<n>We introduce a diagnostic benchmark, GrinningFace, an emoji tabletop manipulation task.<n>We investigate the effects of parameter-efficient fine-tuning, VLM freezing, co-training, predicting discretized actions, and predicting latent actions.
  arXiv  Detail & Related papers  (2025-11-10T01:58:02Z)
• From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models [5.660635614478238]
  Vision-Language-Action (VLA) models promise to produce versatile, "generalist" robot policies.<n>Traditional imitation learning benchmarks are unsuitable due to the lack of language instructions.<n>We introduce a unified suite of 50 simulation-based tasks across 10 subcategories spanning language instruction, vision, and objects.
  arXiv  Detail & Related papers  (2025-06-11T16:52:18Z)
• Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better [58.559985503802054]
  Vision-language-action (VLA) models combine end-to-end learning with transfer of semantic knowledge from web-scale vision-language model (VLM) training.<n>The most powerful VLMs have tens or hundreds of billions of parameters, presenting an obstacle to real-time inference.<n>Recent VLA models have used specialized modules for efficient continuous control, such as action experts or continuous output heads.<n>We show that naively including such experts significantly harms both training speed and knowledge transfer.
  arXiv  Detail & Related papers  (2025-05-29T17:40:09Z)
• MACAROON: Training Vision-Language Models To Be Your Engaged Partners [95.32771929749514]
  Large vision-language models (LVLMs) generate detailed responses even when questions are ambiguous or unlabeled. In this study, we aim to shift LVLMs from passive answer providers to proactive engaged partners. We introduce MACAROON, self-iMaginAtion for ContrAstive pReference OptimizatiON, which instructs LVLMs to autonomously generate contrastive response pairs for unlabeled questions.
  arXiv  Detail & Related papers  (2024-06-20T09:27:33Z)
• SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledge [56.772051051558215]
  Large vision-language models (LVLMs) are ignorant of the up-to-date knowledge, such as LLaVA series, because they cannot be updated frequently. We propose a plug-and-play framework, for augmenting existing LVLMs in handling visual question answering (VQA) about up-to-date knowledge, dubbed SearchLVLMs.
  arXiv  Detail & Related papers  (2024-05-23T13:32:07Z)
• ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models [27.5219975853389]
  We find that pre-trained vision-and-language models (VLMs) and large language models (LLMs) are good at different kinds of visual commonsense reasoning problems. For problems where the goal is to infer conclusions beyond image content,VLMs face difficulties, while LLMs, given sufficient visual evidence, can use commonsense to infer the answer well.
  arXiv  Detail & Related papers  (2023-10-09T17:10:35Z)
• LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models [55.304181390027274]
  This paper presents a comprehensive evaluation of publicly available large multimodal models by building a LVLM evaluation Hub (LVLM-eHub) Our LVLM-eHub consists of $8$ representative LVLMs such as InstructBLIP and MiniGPT-4, which are thoroughly evaluated by a quantitative capability evaluation and an online arena platform. The study reveals several innovative findings. First, instruction-tuned LVLM with massive in-domain data such as InstructBLIP heavily overfits many existing tasks, generalizing poorly in the open-world scenario.
  arXiv  Detail & Related papers  (2023-06-15T16:39:24Z)
• Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models [76.410400238974]
  We propose TTA with feedback to rectify the model output and prevent the model from becoming blindly confident. A CLIP model is adopted as the reward model during TTA and provides feedback for the VLM. The proposed textitreinforcement learning with CLIP feedback(RLCF) framework is highly flexible and universal.
  arXiv  Detail & Related papers  (2023-05-29T11:03:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.