VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward

• URL: http://arxiv.org/abs/2603.26599v1
• Date: Fri, 27 Mar 2026 16:57:51 GMT
• Title: VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward
• Abstract summary: VGGRPO is a latent geometry-guided framework for geometry-aware video post-training.<n>It stitches video diffusion latents to geometry foundation models, enabling direct decoding of scene geometry from the latent space.<n>It improves camera stability, geometry consistency, and overall quality while eliminating costly VAE decoding.
• Score: 14.297118875382871
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale video diffusion models achieve impressive visual quality, yet often fail to preserve geometric consistency. Prior approaches improve consistency either by augmenting the generator with additional modules or applying geometry-aware alignment. However, architectural modifications can compromise the generalization of internet-scale pretrained models, while existing alignment methods are limited to static scenes and rely on RGB-space rewards that require repeated VAE decoding, incurring substantial compute overhead and failing to generalize to highly dynamic real-world scenes. To preserve the pretrained capacity while improving geometric consistency, we propose VGGRPO (Visual Geometry GRPO), a latent geometry-guided framework for geometry-aware video post-training. VGGRPO introduces a Latent Geometry Model (LGM) that stitches video diffusion latents to geometry foundation models, enabling direct decoding of scene geometry from the latent space. By constructing LGM from a geometry model with 4D reconstruction capability, VGGRPO naturally extends to dynamic scenes, overcoming the static-scene limitations of prior methods. Building on this, we perform latent-space Group Relative Policy Optimization with two complementary rewards: a camera motion smoothness reward that penalizes jittery trajectories, and a geometry reprojection consistency reward that enforces cross-view geometric coherence. Experiments on both static and dynamic benchmarks show that VGGRPO improves camera stability, geometry consistency, and overall quality while eliminating costly VAE decoding, making latent-space geometry-guided reinforcement an efficient and flexible approach to world-consistent video generation.

Related papers

• Visual Geometry Foundation-Aware Gaussians for Single-Frame Surround-View Driving Reconstruction [44.54907293631917]
  Single-frame surround-view reconstruction faces severe geometric instability and rendering artifacts due to minimal inter-camera overlap.<n>We propose VGGD, a visual geometry foundation-aware 3D Gaussian Splatting framework for feed-forward surround-view driving reconstruction.
  arXiv  Detail & Related papers  (2026-08-11T09:04:33Z)
• Ground4D: Consistency-Aware 4D Reconstruction from Monocular Video [69.94904173314505]
  We propose Ground4D, a geometry-grounded framework built on two stages.<n>First, we reconstruct multi-view-consistent 3D geometry and camera poses from monocular video.<n>Second, we conduct geometry-consistency-aware refinement via dynamic Gaussian Splatting.
  arXiv  Detail & Related papers  (2026-06-27T09:27:42Z)
• IVGT: Implicit Visual Geometry Transformer for Neural Scene Representation [76.36174247570716]
  We propose IVGT, an Implicit Visual Geometry Transformer that implicitly models continuous and coherent geometry from pose-free multi-view images.<n> IVGT learns a continuous neural scene representation in a canonical coordinate system and supports continuous spatial queries at any 3D positions.<n>It allows direct extraction of continuous and coherent surface geometry, enabling rendering of RGB images, depth maps, and surface normal maps from arbitrary viewpoints.
  arXiv  Detail & Related papers  (2026-05-15T17:59:57Z)
• Beyond Gaussian Bottlenecks: Topologically Aligned Encoding of Vision-Transformer Feature Spaces [18.19675060489249]
  We propose S$2$VAE, a geometry-first latent learning framework that focuses on compressing and representing the latent 3D state of a scene.<n>In depth estimation, camera pose recovery, and point cloud reconstruction, we show that geometry-aligned hyperspherical latents consistently outperform conventional Gaussian bottlenecks.
  arXiv  Detail & Related papers  (2026-04-30T17:12:31Z)
• World-R1: Reinforcing 3D Constraints for Text-to-Video Generation [36.18760315762835]
  We propose World-R1, a framework that aligns video generation with 3D constraints through reinforcement learning.<n>We optimize the model using feedback from pre-trained 3D foundation models and vision-language models.<n>Our approach significantly enhances 3D consistency while preserving the original visual quality of the foundation model.
  arXiv  Detail & Related papers  (2026-04-27T17:59:56Z)
• Repurposing Geometric Foundation Models for Multi-view Diffusion [54.65956050951508]
  We propose a framework that repurposes the geometrically consistent feature space of geometric foundation models as the latent space for multi-view diffusion.<n>Our experiments demonstrate that GLD outperforms both VAE and RAE on 2D image quality and 3D consistency metrics, while accelerating training by more than 4.4x compared to the VAE latent space.
  arXiv  Detail & Related papers  (2026-03-23T17:57:05Z)
• VIGOR: VIdeo Geometry-Oriented Reward for Temporal Generative Alignment [15.619170225414571]
  Video diffusion models lack explicit geometric supervision during training, leading to inconsistency artifacts.<n>We propose a geometry-based reward model that leverages pretrained geometric foundation models to evaluate multi-view consistency.<n>Our approach conducts error computation in a pointwise fashion, yielding a more physically grounded and robust error metric.
  arXiv  Detail & Related papers  (2026-03-17T09:04:10Z)
• RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space [51.441415833480505]
  RAYNOVA is a multiview world model for driving scenarios that employs a dual-causal autoregressive framework.<n>It constructs an isotropic-temporal representation across views, frames, and scales based on relative Plcker-ray positional encoding.
  arXiv  Detail & Related papers  (2026-02-24T08:41:40Z)
• Scalable Adaptation of 3D Geometric Foundation Models via Weak Supervision from Internet Video [76.32954467706581]
  We propose SAGE, a framework for Scalable Adaptation of GEometric foundation models from raw video streams.<n>We use a hierarchical mining pipeline to transform videos into training trajectories and hybrid supervision.<n>Experiments show that SAGE significantly enhances zero-shot generalization, reducing Chamfer Distance by 20-42% on unseen benchmarks.
  arXiv  Detail & Related papers  (2026-02-08T09:53:21Z)
• GeoVideo: Introducing Geometric Regularization into Video Generation Model [46.38507581500745]
  We introduce geometric regularization losses into video generation by augmenting latent diffusion models with per-frame depth prediction.<n>Our method bridges the gap between appearance generation and 3D structure modeling, leading to improved structural coherence-temporal shape, consistency, and physical plausibility.
  arXiv  Detail & Related papers  (2025-12-03T05:11:57Z)
• GeoWorld: Unlocking the Potential of Geometry Models to Facilitate High-Fidelity 3D Scene Generation [68.02988074681427]
  Previous works leveraging video models for image-to-3D scene generation tend to suffer from geometric distortions and blurry content.<n>In this paper, we renovate the pipeline of image-to-3D scene generation by unlocking the potential of geometry models.<n>Our GeoWorld can generate high-fidelity 3D scenes from a single image and a given camera trajectory, outperforming prior methods both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2025-11-28T13:55:45Z)
• Epipolar Geometry Improves Video Generation Models [73.44978239787501]
  3D-consistent video generation could significantly impact numerous downstream applications in generation and reconstruction tasks.<n>We explore how epipolar geometry constraints improve modern video diffusion models.<n>By bridging data-driven deep learning with classical geometric computer vision, we present a practical method for generating spatially consistent videos.
  arXiv  Detail & Related papers  (2025-10-24T16:21:37Z)
• UniGeo: Taming Video Diffusion for Unified Consistent Geometry Estimation [63.90470530428842]
  In this work, we demonstrate that, through appropriate design and fine-tuning, the intrinsic consistency of video generation models can be effectively harnessed for consistent geometric estimation.<n>Our results achieve superior performance in predicting global geometric attributes in videos and can be directly applied to reconstruction tasks.
  arXiv  Detail & Related papers  (2025-05-30T12:31:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.