UniCoD: Enhancing Robot Policy via Unified Continuous and Discrete Representation Learning

• URL: http://arxiv.org/abs/2510.10642v2
• Date: Tue, 04 Nov 2025 09:35:36 GMT
• Title: UniCoD: Enhancing Robot Policy via Unified Continuous and Discrete Representation Learning
• Authors: Jianke Zhang, Yucheng Hu, Yanjiang Guo, Xiaoyu Chen, Yichen Liu, Wenna Chen, Chaochao Lu, Jianyu Chen,
• Abstract summary: Building generalist robot policies that can handle diverse tasks in open-ended environments is a central challenge in robotics.<n>To leverage knowledge from large-scale pretraining, prior work has typically built generalist policies either on top of vision-language understanding models (VLMs) or generative models.<n>Recent unified models of generation and understanding have demonstrated strong capabilities in both comprehension and generation through large-scale pretraining.<n>We introduce UniCoD, which acquires the ability to dynamically model high-dimensional visual features through pretraining on over 1M internet-scale instructional manipulation videos.
• Score: 22.84748754972181
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Building generalist robot policies that can handle diverse tasks in open-ended environments is a central challenge in robotics. To leverage knowledge from large-scale pretraining, prior work (VLA) has typically built generalist policies either on top of vision-language understanding models (VLMs) or generative models. However, both semantic understanding from vision-language pretraining and visual dynamics modeling from visual-generation pretraining are crucial for embodied robots. Recent unified models of generation and understanding have demonstrated strong capabilities in both comprehension and generation through large-scale pretraining. We posit that robotic policy learning can likewise benefit from the combined strengths of understanding, planning, and continuous future representation learning. Building on this insight, we introduce UniCoD, which acquires the ability to dynamically model high-dimensional visual features through pretraining on over 1M internet-scale instructional manipulation videos. Subsequently, UniCoD is fine-tuned on data collected from the robot embodiment, enabling the learning of mappings from predictive representations to action tokens. Extensive experiments show our approach consistently outperforms baseline methods in terms of 9\% and 12\% across simulation environments and real-world out-of-distribution tasks.

Related papers

• mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs [5.109732854501585]
  We introduce mimic-video, a novel Video-Action Model (VAM) that pairs a pretrained Internet-scale video model with a flow matching-based action decoder conditioned on its latent representations.<n>Our approach achieves state-of-the-art performance on simulated and real-world robotic manipulation tasks, improving sample efficiency by 10x and convergence speed by 2x compared to traditional VLA architectures.
  arXiv  Detail & Related papers  (2025-12-17T18:47:31Z)
• Large Video Planner Enables Generalizable Robot Control [117.49024534548319]
  General-purpose robots require decision-making models that generalize across diverse tasks and environments.<n>Recent works build robot foundation models by extending multimodal large language models (LMs) with action outputs, creating vision--action (VLA) systems.<n>We explore an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models.
  arXiv  Detail & Related papers  (2025-12-17T18:35:54Z)
• VideoVLA: Video Generators Can Be Generalizable Robot Manipulators [86.70243911696616]
  Generalization in robot manipulation is essential for deploying robots in open-world environments.<n>We present VideoVLA, a simple approach that explores the potential of transforming large video generation models into robotic VLA manipulators.
  arXiv  Detail & Related papers  (2025-12-07T18:57:15Z)
• Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations [26.678553477485362]
  We present a framework that better preserves pretrained features while adapting them for robot manipulation.<n>Our approach introduces three components: (i) a dual-encoder design with one frozen vision to retain pretrained features and another trainable for task adaptation, (ii) a string-based action tokenizer that casts continuous actions into character sequences aligned with the model's pretraining domain, and (iii) a co-training strategy that combines robot demonstrations with vision-language datasets emphasizing spatial reasoning and affordances.
  arXiv  Detail & Related papers  (2025-09-14T20:08:56Z)
• Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations [19.45821593625599]
  Video diffusion models (VDMs) demonstrate the ability to predict future frames and showcase a strong understanding of physical world.<n>We propose the Video Prediction Policy (VPP), which learns implicit inverse dynamics model conditioned on predicted future representations inside VDMs.<n>VPP achieves a 18.6% relative improvement on the Calvin ABC-D generalization benchmark compared to the previous state-of-the-art.
  arXiv  Detail & Related papers  (2024-12-19T12:48:40Z)
• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• Latent Action Pretraining from Videos [156.88613023078778]
  We introduce Latent Action Pretraining for general Action models (LAPA)<n>LAPA is an unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels.<n>We propose a method to learn from internet-scale videos that do not have robot action labels.
  arXiv  Detail & Related papers  (2024-10-15T16:28:09Z)
• KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data [45.25288643161976]
  We propose Keypoint Affordance Learning from Imagined Environments (KALIE) for robotic control in a scalable manner. Instead of directly producing motor commands, KALIE controls the robot by predicting point-based affordance representations. We demonstrate that KALIE can learn to robustly solve new manipulation tasks with unseen objects given only 50 example data points.
  arXiv  Detail & Related papers  (2024-09-21T08:45:16Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations.<n>First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets.<n>We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)
• Learning Manipulation by Predicting Interaction [85.57297574510507]
  We propose a general pre-training pipeline that learns Manipulation by Predicting the Interaction. The experimental results demonstrate that MPI exhibits remarkable improvement by 10% to 64% compared with previous state-of-the-art in real-world robot platforms.
  arXiv  Detail & Related papers  (2024-06-01T13:28:31Z)
• Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methods [14.780597545674157]
  We investigate the effects of visual pre-training strategies on robot manipulation tasks from three fundamental perspectives. We propose a visual pre-training scheme for robot manipulation termed Vi-PRoM, which combines self-supervised learning and supervised learning.
  arXiv  Detail & Related papers  (2023-08-07T14:24:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.