PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation
Abstract Overview
PointDiT formulates monocular geometry estimation as conditional pixel-space diffusion over dense 3D point maps rather than deterministic regression or latent diffusion. The model is intentionally simple: a plain Vision Transformer operates directly on raw point-map patches and is conditioned on frozen DINOv3 image tokens, while training uses flow matching with clean point-map prediction. The paper also introduces task-specific handling for point-map normalization and sky regions so indoor and outdoor data can be trained jointly. Experiments evaluate zero-shot transfer from synthetic-only training to seven real-world datasets and analyze single-step, multi-step, and ablated variants.
Novelty
The main novelty is applying a minimalist pixel-space diffusion transformer directly to monocular point-map prediction, avoiding both VAE-based latent tokenization and the hybrid architectures common in prior geometry systems. The work further argues that clean-data (x-prediction) is crucial for 3D point-map diffusion and shows that this setup can support competitive one-step inference.
Results
On seven real-world evaluation datasets, PointDiT-H achieves the best reported depth metrics in the comparison table (Rel_d 2.75 and δ1_d 98.54 with multi-step inference) and the highest boundary sharpness BF1 of 10.49, while MoGe remains slightly better on point-map Rel_p (4.21 vs. 4.40). Even one-step PointDiT-H already exceeds prior methods on BF1 (9.79 vs. best baseline 9.41) and is much faster than GeometryCrafter at single-step inference (72 ms vs. 1,178 ms). Additional diffusion steps mainly refine boundaries, and controlled experiments show the generative formulation improves BF1 over a matched deterministic regressor (13.92 vs. 10.90).
Key Points
- PointDiT uses a plain ViT to diffuse directly in raw 3D point-map space, conditioned by multi-layer DINOv3 features, instead of relying on latent VAEs or complex hybrid geometry architectures.
- The method is trained largely with flow matching plus a lightweight relative point loss, and its ablations show that clean point-map prediction (x-prediction) is critical while direct velocity prediction performs poorly.
- Synthetic-only training transfers zero-shot to seven real-world datasets, where the model delivers especially strong geometric boundary quality and supports both efficient one-step inference and further multi-step refinement.