Video Generation Models are General-Purpose Vision Learners

• URL: http://arxiv.org/abs/2607.09024v1
• Date: Fri, 10 Jul 2026 01:09:06 GMT
• Title: Video Generation Models are General-Purpose Vision Learners
• Abstract summary: We argue that large-scale text-to-video generation serves as a strong pre-training paradigm for computer vision.<n>We introduce GenCeption, which leverages a pre-trained video video diffusion backbone to define a feed-forward perception model.<n> Empirical results demonstrate that GenCeption achieves state-of-the-art performance across a diverse suite of tasks.
• Score: 62.28137589984863
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Driven by next-token prediction, NLP shifted from task-specific models into powerful generalist foundation models. What, then, is the equivalent catalyst needed to achieve a general-purpose model in computer vision? In this paper, we contend that large-scale text-to-video generation serves as a strong pre-training paradigm for computer vision, providing the necessary spatiotemporal priors, vision-language alignment, and scalability required for general visual intelligence. We introduce GenCeption, which leverages a pre-trained video generative diffusion backbone to define a feed-forward perception model, capable of performing various vision tasks steered by text instructions. Empirical results demonstrate that GenCeption achieves state-of-the-art performance across a diverse suite of tasks, including depth, surface normal, and camera pose estimation, expression-referring segmentation, and 3D keypoint prediction, often matching or surpassing specialized models (e.g. DepthAnything3, SAM3, D4RT, VGGT-Omega, Sapiens, David, Genmo, and Lotus-2). Furthermore, the video generative pretrained backbone outperforms alternative pretraining paradigms (e.g., V-JEPA, and Video MAE) under comparable settings. Importantly, GenCeption exhibits preliminary data and model scaling properties along with exceptional data efficiency, where it achieves comparable performance with leading models like D4RT and VGGT-Omega with 7 to 500 less training data. Finally, GenCeption also exhibits intriguing emergent behaviors: a model trained exclusively on synthetic human videos generalizes to real-world footage and out-of-distribution object categories (e.g., animals and robots). These findings suggest that video generation is not merely a synthesis tool, but a foundational path toward generalist vision intelligence for the physical world. Project page: https://genception.github.io

Related papers

• From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data [71.22409934108924]
  Human videos are abundant and capture rich interactions, providing diverse semantic and physical cues for real-world manipulation.<n>This survey provides a unified view of how human videos are transformed into effective knowledge for Vision-Language-Action (VLA) models.<n>We highlight three key open challenges in this area: structuring unstructured videos into training-ready episodes, grounding video-derived supervision into robot-executable actions under embodiment and viewpoint heterogeneity, and designing evaluation protocols that better predict real-world deployment performance and transfer efficiency.
  arXiv  Detail & Related papers  (2026-05-18T06:19:16Z)
• Image Generators are Generalist Vision Learners [71.11980587450198]
  We show that image generation training serves a role similar to language understanding and reasoning pretraining.<n>We introduce Vision Banana, a generalist model built by instruction-tuning Nano Banana Pro (NBP) on a mixture of its original training data and vision task data.<n>Our model achieves SOTA results on a variety of vision tasks involving both 2D and 3D understanding, beating or rivaling zero-shot domain-specialists.
  arXiv  Detail & Related papers  (2026-04-22T08:23:48Z)
• Robotic Manipulation is Vision-to-Geometry Mapping ($f(v) \rightarrow G$): Vision-Geometry Backbones over Language and Video Models [65.05130114320734]
  We argue that the foundation for generalizable robotic control should be a vision-geometry backbone, rather than the widely adopted vision-language or video models.<n>We propose the Vision-Geometry-Action model, which directly conditions action generation on pretrained native 3D representations.<n> Specifically, VGA replaces conventional language or video backbones with a pretrained 3D world model, establishing a seamless vision-to-geometry mapping.
  arXiv  Detail & Related papers  (2026-04-14T15:57:16Z)
• Disentangled Robot Learning via Separate Forward and Inverse Dynamics Pretraining [28.30092786035367]
  DeFI is a novel framework that Decouples visual Forward and Inverse dynamics pretraining to exploit respective data sources.<n>We introduce the General Forward Dynamics Model (GFDM), pretrained on diverse human and robot videos for future prediction, and the General Inverse Dynamics Model (GIDM), trained via self-supervised learning to infer latent actions from unlabeled video transitions.<n>Experiments on CALVIN ABC-D and SimplerEnv demonstrate state-of-the-art performance, with DeFI achieving an average task length of 4.51 for CALVIN, 51.2% success rate on SimplerEnv-Frac
  arXiv  Detail & Related papers  (2026-03-27T17:20:10Z)
• THFM: A Unified Video Foundation Model for 4D Human Perception and Beyond [22.66024130777808]
  We present THFM, a unified video foundation model for human-centric perception.<n>Our model is on-par or surpassing state-of-the-art specialized models on a variety of benchmarks.<n>For example, our model trained on videos with a single human in the scene generalizes to multiple humans and other object classes such as anthropomorphic characters and animals.
  arXiv  Detail & Related papers  (2026-03-26T20:28:40Z)
• From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Models [65.0487600936788]
  Video Diffusion Models (VDMs) have emerged as powerful generative tools capable of synthesizing high-quality content.<n>We argue that VDMs naturally push to probe structured representations and an implicit understanding of the visual world.<n>Our method transforms each task into a visual transition, enabling the training of LoRA weights on short input- sequences.
  arXiv  Detail & Related papers  (2025-06-08T20:52:34Z)
• Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations [19.45821593625599]
  Video diffusion models (VDMs) demonstrate the ability to predict future frames and showcase a strong understanding of physical world.<n>We propose the Video Prediction Policy (VPP), which learns implicit inverse dynamics model conditioned on predicted future representations inside VDMs.<n>VPP achieves a 18.6% relative improvement on the Calvin ABC-D generalization benchmark compared to the previous state-of-the-art.
  arXiv  Detail & Related papers  (2024-12-19T12:48:40Z)
• Predicting Long-horizon Futures by Conditioning on Geometry and Time [49.86180975196375]
  We explore the task of generating future sensor observations conditioned on the past. We leverage the large-scale pretraining of image diffusion models which can handle multi-modality. We create a benchmark for video prediction on a diverse set of videos spanning indoor and outdoor scenes.
  arXiv  Detail & Related papers  (2024-04-17T16:56:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.