Video Generative Models as Geometry Learner
Abstract Overview
This paper proposes GeoNeXt, a unified framework for monocular geometry estimation that repurposes a pretrained video generative model for depth and surface normal prediction. Instead of adapting an image diffusion model for each geometry task independently, it formulates geometry as next-frame generation, treating depth and surface normals as subsequent frames conditioned on an input RGB image. The method jointly denoises image, depth, and normal latents so that appearance and geometry are reconstructed along a shared trajectory, requiring only lightweight adaptation of the pretrained video model. Trained on roughly 59K synthetic samples from Hypersim and Virtual KITTI 2, GeoNeXt demonstrates strong zero-shot generalization across diverse real-world depth and surface normal benchmarks.
Novelty
The key novelty is leveraging a pretrained video diffusion model as a unified geometry learner by casting depth and surface normal estimation as next-frame generation. This enables the reuse of temporal cross-frame priors for joint image-geometry modeling without requiring heavy architectural modifications or separate task-specific networks.
Results
Across five zero-shot depth benchmarks, GeoNeXt outperforms the unified generative baseline GeoWizard on all reported datasets despite using only 59K training samples versus 208K, achieving AbsRel/δ1 of 8.2/92.6 on KITTI and 5.6/97.2 on ETH3D compared to GeoWizard's 14.4/82.0 and 6.8/95.8. On five surface-normal benchmarks, it consistently improves over GeoWizard, including mean angular error/11.25° accuracy of 16.4/69.2 on iBims-1 and 33.0/21.5 on Sintel. Ablations confirm that jointly reconstructing the input image and co-training depth with normals are crucial for performance, while the model remains robust to the reconstruction order of depth and normals.
Key Points
- GeoNeXt formulates monocular depth and surface normal estimation as image-to-video next-frame generation, enabling unified modeling of image appearance and geometry within a pretrained video diffusion model.
- The method fine-tunes only the denoising U-Net with lightweight adaptation, training on approximately 59K synthetic samples while generalizing zero-shot across diverse real-world benchmarks.
- Experimental results show consistent gains over prior unified generative models and competitive accuracy against large-scale discriminative baselines, supported by ablation studies verifying the joint generation design.