Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations

• URL: http://arxiv.org/abs/2509.11417v2
• Date: Wed, 17 Sep 2025 02:41:34 GMT
• Title: Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations
• Authors: Shresth Grover, Akshay Gopalkrishnan, Bo Ai, Henrik I. Christensen, Hao Su, Xuanlin Li,
• Abstract summary: We present a framework that better preserves pretrained features while adapting them for robot manipulation.<n>Our approach introduces three components: (i) a dual-encoder design with one frozen vision to retain pretrained features and another trainable for task adaptation, (ii) a string-based action tokenizer that casts continuous actions into character sequences aligned with the model's pretraining domain, and (iii) a co-training strategy that combines robot demonstrations with vision-language datasets emphasizing spatial reasoning and affordances.
• Score: 26.678553477485362
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language-action (VLA) models finetuned from vision-language models (VLMs) hold the promise of leveraging rich pretrained representations to build generalist robots across diverse tasks and environments. However, direct fine-tuning on robot data often disrupts these representations and limits generalization. We present a framework that better preserves pretrained features while adapting them for robot manipulation. Our approach introduces three components: (i) a dual-encoder design with one frozen vision encoder to retain pretrained features and another trainable for task adaptation, (ii) a string-based action tokenizer that casts continuous actions into character sequences aligned with the model's pretraining domain, and (iii) a co-training strategy that combines robot demonstrations with vision-language datasets emphasizing spatial reasoning and affordances. Evaluations in simulation and on real robots show that our method improves robustness to visual perturbations, generalization to novel instructions and environments, and overall task success compared to baselines.

Related papers

• Universal Pose Pretraining for Generalizable Vision-Language-Action Policies [83.39008378156647]
  Existing Vision-Language-Action (VLA) models often suffer from feature collapse and low training efficiency.<n>We propose Pose-VLA, a decoupled paradigm that separates VLA training into a pre-training phase for extracting universal 3D spatial priors.<n>Our framework follows a two-stage pre-training pipeline, establishing fundamental spatial grounding via poses followed by motion alignment.
  arXiv  Detail & Related papers  (2026-02-23T11:00:08Z)
• mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs [5.109732854501585]
  We introduce mimic-video, a novel Video-Action Model (VAM) that pairs a pretrained Internet-scale video model with a flow matching-based action decoder conditioned on its latent representations.<n>Our approach achieves state-of-the-art performance on simulated and real-world robotic manipulation tasks, improving sample efficiency by 10x and convergence speed by 2x compared to traditional VLA architectures.
  arXiv  Detail & Related papers  (2025-12-17T18:47:31Z)
• VideoVLA: Video Generators Can Be Generalizable Robot Manipulators [86.70243911696616]
  Generalization in robot manipulation is essential for deploying robots in open-world environments.<n>We present VideoVLA, a simple approach that explores the potential of transforming large video generation models into robotic VLA manipulators.
  arXiv  Detail & Related papers  (2025-12-07T18:57:15Z)
• Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations [76.79742393097358]
  Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics.<n>Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics.<n>Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task.
  arXiv  Detail & Related papers  (2025-11-27T18:50:21Z)
• iFlyBot-VLA Technical Report [25.330744626382977]
  We introduce iFlyBot-VLA, a large-scale Vision-Language-Action (VLA) model trained under a novel framework.<n>The main contributions are listed as follows: (1) a latent action model thoroughly trained on large-scale human and robotic manipulation videos; (2) a dual-level action representation framework that jointly supervises both the Vision-Language Model (VLM) and the action expert during training; and (3) a mixed training strategy that combines robot trajectory data with general QA and spatial QA datasets.
  arXiv  Detail & Related papers  (2025-11-01T06:24:56Z)
• UniCoD: Enhancing Robot Policy via Unified Continuous and Discrete Representation Learning [22.84748754972181]
  Building generalist robot policies that can handle diverse tasks in open-ended environments is a central challenge in robotics.<n>To leverage knowledge from large-scale pretraining, prior work has typically built generalist policies either on top of vision-language understanding models (VLMs) or generative models.<n>Recent unified models of generation and understanding have demonstrated strong capabilities in both comprehension and generation through large-scale pretraining.<n>We introduce UniCoD, which acquires the ability to dynamically model high-dimensional visual features through pretraining on over 1M internet-scale instructional manipulation videos.
  arXiv  Detail & Related papers  (2025-10-12T14:54:19Z)
• RoboAct-CLIP: Video-Driven Pre-training of Atomic Action Understanding for Robotics [22.007302996282085]
  This paper presents a temporal-decoupling fine-tuning strategy based on Contrastive Language-Image Pretraining (CLIP) architecture.<n>Results in simulated environments demonstrate that the RoboAct-CLIP pretrained model achieves a 12% higher success rate than baseline Visual Language Models.
  arXiv  Detail & Related papers  (2025-04-02T19:02:08Z)
• Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy [73.75271615101754]
  We present Dita, a scalable framework that leverages Transformer architectures to directly denoise continuous action sequences.<n>Dita employs in-context conditioning -- enabling fine-grained alignment between denoised actions and raw visual tokens from historical observations.<n>Dita effectively integrates cross-embodiment datasets across diverse camera perspectives, observation scenes, tasks, and action spaces.
  arXiv  Detail & Related papers  (2025-03-25T15:19:56Z)
• SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model [45.03115608632622]
  spatial understanding is the keypoint in robot manipulation.<n>We propose SpatialVLA to explore effective spatial representations for the robot foundation model.<n>We show the proposed Adaptive Action Grids offer a new and effective way to fine-tune the pre-trained SpatialVLA model for new simulation and real-world setups.
  arXiv  Detail & Related papers  (2025-01-27T07:34:33Z)
• Diffusion Transformer Policy [48.50988753948537]
  We propose a large multi-modal diffusion transformer, dubbed as Diffusion Transformer Policy, to model continuous end-effector actions.<n>By leveraging the scaling capability of transformers, the proposed approach can effectively model continuous end-effector actions across large diverse robot datasets.
  arXiv  Detail & Related papers  (2024-10-21T12:43:54Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [140.48218261864153]
  We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control. Our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training.
  arXiv  Detail & Related papers  (2023-07-28T21:18:02Z)
• Instruction-Following Agents with Multimodal Transformer [95.70039658112873]
  We propose a simple yet effective model for robots to solve instruction-following tasks in vision-based environments. Our method consists of a multimodal transformer that encodes visual observations and language instructions. We show that this unified transformer model outperforms all state-of-the-art pre-trained or trained-from-scratch methods in both single-task and multi-task settings.
  arXiv  Detail & Related papers  (2022-10-24T17:46:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.