RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulation

• URL: http://arxiv.org/abs/2601.05241v1
• Date: Thu, 08 Jan 2026 18:59:22 GMT
• Title: RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulation
• Authors: Boyang Wang, Haoran Zhang, Shujie Zhang, Jinkun Hao, Mingda Jia, Qi Lv, Yucheng Mao, Zhaoyang Lyu, Jia Zeng, Xudong Xu, Jiangmiao Pang,
• Abstract summary: We build a scalable pipeline to curate a visual identity pool from large datasets.<n>Using our augmented manipulation data to train downstream vision-language-action and visuomotor policy models yields consistent performance gains in both simulation and real-robot settings.
• Score: 38.65727896970541
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The diversity, quantity, and quality of manipulation data are critical for training effective robot policies. However, due to hardware and physical setup constraints, collecting large-scale real-world manipulation data remains difficult to scale across diverse environments. Recent work uses text-prompt conditioned image diffusion models to augment manipulation data by altering the backgrounds and tabletop objects in the visual observations. However, these approaches often overlook the practical need for multi-view and temporally coherent observations required by state-of-the-art policy models. Further, text prompts alone cannot reliably specify the scene setup. To provide the diffusion model with explicit visual guidance, we introduce visual identity prompting, which supplies exemplar images as conditioning inputs to guide the generation of the desired scene setup. To this end, we also build a scalable pipeline to curate a visual identity pool from large robotics datasets. Using our augmented manipulation data to train downstream vision-language-action and visuomotor policy models yields consistent performance gains in both simulation and real-robot settings.

Related papers

• Future Optical Flow Prediction Improves Robot Control & Video Generation [100.87884718953099]
  We introduce FOFPred, a novel optical flow forecasting model featuring a unified Vision-Language Model (VLM) and Diffusion architecture.<n>Our model is trained on web-scale human activity data-a highly scalable but unstructured source.<n> Evaluations across robotic manipulation and video generation under language-driven settings establish the cross-domain versatility of FOFPred.
  arXiv  Detail & Related papers  (2026-01-15T18:49:48Z)
• Invariance Co-training for Robot Visual Generalization [72.88252238231269]
  Reasoning from diverse observations is a fundamental capability for generalist robot policies to operate in a wide range of environments.<n>We show that co-training on diverse data improves performance by 18 percent over existing generative augmentation methods.
  arXiv  Detail & Related papers  (2025-12-04T20:08:46Z)
• EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfer [35.27100635173712]
  Vision-language-action (VLA) models increasingly rely on diverse training data to achieve robust generalization.<n>We propose Embodied Manipulation Media Adaptation (EMMA), a VLA policy enhancement framework that integrates a generative data engine with an effective training pipeline.<n>DreamTransfer enables text-controlled visual editing of robot videos, transforming foreground, background, and lighting conditions without compromising 3D structure or geometrical plausibility.<n>AdaMix is a hard-sample-aware training strategy that dynamically reweights training batches to focus optimization on perceptually or kinematically challenging samples.
  arXiv  Detail & Related papers  (2025-09-26T14:34:44Z)
• ASMR: Augmenting Life Scenario using Large Generative Models for Robotic Action Reflection [21.75681306780917]
  This paper introduces a novel framework focusing on data augmentation in robotic assistance scenarios.<n>It involves leveraging a sophisticated large language model to simulate potential conversations and environmental contexts.<n>The additionally generated data serves to refine the latest multimodal models, enabling them to more accurately determine appropriate actions.
  arXiv  Detail & Related papers  (2025-06-16T19:58:54Z)
• ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation? [23.73409728939164]
  This paper introduces ManipGPT, a framework designed to predict optimal interaction areas for articulated objects.<n>We create a dataset of 9.9k simulated and real images to bridge the visual sim-to-real gap.<n>We significantly improve part-level affordance segmentation, adapting the model's in-context segmentation capabilities to robot manipulation scenarios.
  arXiv  Detail & Related papers  (2024-12-13T11:22:01Z)
• SKT: Integrating State-Aware Keypoint Trajectories with Vision-Language Models for Robotic Garment Manipulation [82.61572106180705]
  This paper presents a unified approach using vision-language models (VLMs) to improve keypoint prediction across various garment categories. We created a large-scale synthetic dataset using advanced simulation techniques, allowing scalable training without extensive real-world data. Experimental results indicate that the VLM-based method significantly enhances keypoint detection accuracy and task success rates.
  arXiv  Detail & Related papers  (2024-09-26T17:26:16Z)
• BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation [57.40024206484446]
  We introduce the BEHAVIOR Vision Suite (BVS), a set of tools and assets to generate fully customized synthetic data for systematic evaluation of computer vision models. BVS supports a large number of adjustable parameters at the scene level. We showcase three example application scenarios.
  arXiv  Detail & Related papers  (2024-05-15T17:57:56Z)
• Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks [0.0]
  In this work, we focus on unsupervised vision-language--action mapping in the area of robotic manipulation.<n>We propose a model-invariant training alternative that improves the models' performance in a simulator by up to 55%.<n>Our work thus also sheds light on the potential benefits and limitations of using the current multimodal VAEs for unsupervised learning of robotic motion trajectories.
  arXiv  Detail & Related papers  (2024-04-02T13:25:16Z)
• Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception [0.0]
  Vision-language models (VLMs) have shown powerful capabilities in visual question answering and reasoning tasks. In this paper, we demonstrate a method of aligning the embedding spaces of different modalities to the vision embedding space. We show that using multiple modalities as input improves the VLM's scene understanding and enhances its overall performance in various tasks.
  arXiv  Detail & Related papers  (2023-08-31T06:53:55Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.