Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation

• URL: http://arxiv.org/abs/2508.16512v1
• Date: Fri, 22 Aug 2025 16:35:19 GMT
• Title: Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation
• Abstract summary: We investigate the effects of existing fine-tuning video generation approaches on structured driving datasets.<n>We show that although visual fidelity improves, spatial accuracy in modeling dynamic elements may degrade.
• Score: 17.301302433153467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in video generation have substantially improved visual quality and temporal coherence, making these models increasingly appealing for applications such as autonomous driving, particularly in the context of driving simulation and so-called "world models". In this work, we investigate the effects of existing fine-tuning video generation approaches on structured driving datasets and uncover a potential trade-off: although visual fidelity improves, spatial accuracy in modeling dynamic elements may degrade. We attribute this degradation to a shift in the alignment between visual quality and dynamic understanding objectives. In datasets with diverse scene structures within temporal space, where objects or perspective shift in varied ways, these objectives tend to highly correlated. However, the very regular and repetitive nature of driving scenes allows visual quality to improve by modeling dominant scene motion patterns, without necessarily preserving fine-grained dynamic behavior. As a result, fine-tuning encourages the model to prioritize surface-level realism over dynamic accuracy. To further examine this phenomenon, we show that simple continual learning strategies, such as replay from diverse domains, can offer a balanced alternative by preserving spatial accuracy while maintaining strong visual quality.

Related papers

• Tempered Self-Similarity Alignment for Physically Plausible Video Generation [83.40337664171939]
  Self-similarity loss transforms into probabilistic correspondence and trains the video generative model to align its correspondence with those of the visual foundation model on dynamically changing regions.<n>Our method demonstrates substantial improvements in physical plausibility across diverse interaction scenarios, validating the effectiveness of transferring relational knowledge for video generation.
  arXiv  Detail & Related papers  (2026-05-24T09:28:05Z)
• Kinematics-Aware Latent World Models for Data-Efficient Autonomous Driving [8.391716372682316]
  Vehicle kinematic information is incorporated into the observation encoder to ground latent transitions in physically meaningful motion dynamics.<n> geometry-aware supervision regularizes the RSSM latent state to capture task-relevant spatial structure beyond pixel reconstruction.
  arXiv  Detail & Related papers  (2026-03-07T15:47:54Z)
• DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving [49.11389494068169]
  We present DrivingGen, the first comprehensive benchmark for generative driving world models.<n>DrivingGen combines a diverse evaluation dataset curated from both driving datasets and internet-scale video sources.<n>General models look better but break physics, while driving-specific ones capture motion realistically but lag in visual quality.
  arXiv  Detail & Related papers  (2026-01-04T13:36:21Z)
• ArmGS: Composite Gaussian Appearance Refinement for Modeling Dynamic Urban Environments [22.371417505012566]
  This work focuses on modeling dynamic urban environments for autonomous driving simulation.<n>We propose a new approach named ArmGS that exploits composite driving Gaussian splatting with multi-granularity appearance refinement.<n>This not only models global scene appearance variations between frames and camera viewpoints, but also models local fine-grained photorealistic changes of background and objects.
  arXiv  Detail & Related papers  (2025-07-05T03:54:40Z)
• GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control [50.67481583744243]
  We introduce GeoDrive, which explicitly integrates robust 3D geometry conditions into driving world models.<n>We propose a dynamic editing module during training to enhance the renderings by editing the positions of the vehicles.<n>Our method significantly outperforms existing models in both action accuracy and 3D spatial awareness.
  arXiv  Detail & Related papers  (2025-05-28T14:46:51Z)
• RAGME: Retrieval Augmented Video Generation for Enhanced Motion Realism [73.38167494118746]
  We propose a framework to improve the realism of motion in generated videos.<n>We advocate for the incorporation of a retrieval mechanism during the generation phase.<n>Our pipeline is designed to apply to any text-to-video diffusion model.
  arXiv  Detail & Related papers  (2025-04-09T08:14:05Z)
• Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better [61.381599921020175]
  Temporal consistency is critical in video prediction to ensure that outputs are coherent and free of artifacts.<n>Traditional methods, such as temporal attention and 3D convolution, may struggle with significant object motion.<n>We propose the Tracktention Layer, a novel architectural component that explicitly integrates motion information using point tracks.
  arXiv  Detail & Related papers  (2025-03-25T17:58:48Z)
• InterDyn: Controllable Interactive Dynamics with Video Diffusion Models [50.38647583839384]
  We propose InterDyn, a framework that generates videos of interactive dynamics given an initial frame and a control signal encoding the motion of a driving object or actor.<n>Our key insight is that large video generation models can act as both neurals and implicit physics simulators'', having learned interactive dynamics from large-scale video data.
  arXiv  Detail & Related papers  (2024-12-16T13:57:02Z)
• TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models [75.20168902300166]
  We propose TrackDiffusion, a novel video generation framework affording fine-grained trajectory-conditioned motion control. A pivotal component of TrackDiffusion is the instance enhancer, which explicitly ensures inter-frame consistency of multiple objects. generated video sequences by our TrackDiffusion can be used as training data for visual perception models.
  arXiv  Detail & Related papers  (2023-12-01T15:24:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.