OpenViGA: Video Generation for Automotive Driving Scenes by Streamlining and Fine-Tuning Open Source Models with Public Data

• URL: http://arxiv.org/abs/2509.15479v1
• Date: Thu, 18 Sep 2025 22:54:13 GMT
• Title: OpenViGA: Video Generation for Automotive Driving Scenes by Streamlining and Fine-Tuning Open Source Models with Public Data
• Authors: Björn Möller, Zhengyang Li, Malte Stelzer, Thomas Graave, Fabian Bettels, Muaaz Ataya, Tim Fingscheidt,
• Abstract summary: We present OpenViGA, an open video generation system for automotive driving scenes.<n>For an image size of 256x256 at 4 fps we are able to predict realistic driving scene videos frame-by-frame with only one frame of algorithmic latency.
• Score: 18.7430500677223
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent successful video generation systems that predict and create realistic automotive driving scenes from short video inputs assign tokenization, future state prediction (world model), and video decoding to dedicated models. These approaches often utilize large models that require significant training resources, offer limited insight into design choices, and lack publicly available code and datasets. In this work, we address these deficiencies and present OpenViGA, an open video generation system for automotive driving scenes. Our contributions are: Unlike several earlier works for video generation, such as GAIA-1, we provide a deep analysis of the three components of our system by separate quantitative and qualitative evaluation: Image tokenizer, world model, video decoder. Second, we purely build upon powerful pre-trained open source models from various domains, which we fine-tune by publicly available automotive data (BDD100K) on GPU hardware at academic scale. Third, we build a coherent video generation system by streamlining interfaces of our components. Fourth, due to public availability of the underlying models and data, we allow full reproducibility. Finally, we also publish our code and models on Github. For an image size of 256x256 at 4 fps we are able to predict realistic driving scene videos frame-by-frame with only one frame of algorithmic latency.

Related papers

• Generative Video Matting [57.186684844156595]
  Video matting has traditionally been limited by the lack of high-quality ground-truth data.<n>Most existing video matting datasets provide only human-annotated imperfect alpha and foreground annotations.<n>We introduce a novel video matting approach that can effectively leverage the rich priors from pre-trained video diffusion models.
  arXiv  Detail & Related papers  (2025-08-11T12:18:55Z)
• VaViM and VaVAM: Autonomous Driving through Video Generative Modeling [88.33638585518226]
  We introduce an open-source auto-regressive video model (VaM) and its companion video-action model (VaVAM) to investigate how video pre-training transfers to real-world driving.<n>We evaluate our models in open- and closed-loop driving scenarios, revealing that video-based pre-training holds promise for autonomous driving.
  arXiv  Detail & Related papers  (2025-02-21T18:56:02Z)
• DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT [33.943125216555316]
  We present DrivingWorld, a GPT-style world model for autonomous driving.<n>We propose a next-state prediction strategy to model temporal coherence between consecutive frames.<n>We also propose a novel masking strategy and reweighting strategy for token prediction to mitigate long-term drifting issues.
  arXiv  Detail & Related papers  (2024-12-27T07:44:07Z)
• Movie Gen: A Cast of Media Foundation Models [133.41504332082667]
  We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio.<n>We show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user's image.
  arXiv  Detail & Related papers  (2024-10-17T16:22:46Z)
• DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation [10.296670127024045]
  DriveScape is an end-to-end framework for multi-view, 3D condition-guided video generation. Our Bi-Directional Modulated Transformer (BiMot) ensures precise alignment of 3D structural information. DriveScape excels in video generation performance, achieving state-of-the-art results on the nuScenes dataset with an FID score of 8.34 and an FVD score of 76.39.
  arXiv  Detail & Related papers  (2024-09-09T09:43:17Z)
• DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes [15.506076058742744]
  We propose DreamForge, an advanced diffusion-based autoregressive video generation model tailored for 3D-controllable long-term generation.<n>We introduce perspective guidance and integrate object-wise position encoding to incorporate local 3D correlation.<n>We can autoregressively generate long videos (over 200 frames) using a model trained in short sequences, achieving superior quality compared to the baseline in 16-frame video evaluations.
  arXiv  Detail & Related papers  (2024-09-06T03:09:58Z)
• AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction [88.70116693750452]
  Text-guided video prediction (TVP) involves predicting the motion of future frames from the initial frame according to an instruction. Previous TVP methods make significant breakthroughs by adapting Stable Diffusion for this task. We introduce the Multi-Modal Large Language Model (MLLM) to predict future video states based on initial frames and text instructions.
  arXiv  Detail & Related papers  (2024-06-10T17:02:08Z)
• Streaming Dense Video Captioning [85.70265343236687]
  An ideal model for dense video captioning should be able to handle long input videos, predict rich, detailed textual descriptions. Current state-of-the-art models process a fixed number of downsampled frames, and make a single full prediction after seeing the whole video. We propose a streaming dense video captioning model that consists of two novel components.
  arXiv  Detail & Related papers  (2024-04-01T17:59:15Z)
• VideoCrafter1: Open Diffusion Models for High-Quality Video Generation [97.5767036934979]
  We introduce two diffusion models for high-quality video generation, namely text-to-video (T2V) and image-to-video (I2V) models. T2V models synthesize a video based on a given text input, while I2V models incorporate an additional image input. Our proposed T2V model can generate realistic and cinematic-quality videos with a resolution of $1024 times 576$, outperforming other open-source T2V models in terms of quality.
  arXiv  Detail & Related papers  (2023-10-30T13:12:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.