Geometrically Consistent Multi-View Scene Generation from Freehand Sketches

• URL: http://arxiv.org/abs/2604.14302v1
• Date: Wed, 15 Apr 2026 18:00:45 GMT
• Title: Geometrically Consistent Multi-View Scene Generation from Freehand Sketches
• Abstract summary: Freehand sketches are the most geometrically impoverished input one could offer a multi-view generator.<n>We address three compounding challenges; absent training data, the need for geometric reasoning from distorted 2D input, and cross-view consistency.<n>Our framework synthesizes all views in a single denoising process without requiring reference images, iterative refinement, or per-scene optimisation.
• Score: 58.98194920417429
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We tackle a new problem: generating geometrically consistent multi-view scenes from a single freehand sketch. Freehand sketches are the most geometrically impoverished input one could offer a multi-view generator. They convey scene intent through abstract strokes while introducing spatial distortions that actively conflict with any consistent 3D interpretation. No prior method attempts this; existing multi-view approaches require photographs or text, while sketch-to-3D methods need multiple views or costly per-scene optimisation. We address three compounding challenges; absent training data, the need for geometric reasoning from distorted 2D input, and cross-view consistency, through three mutually reinforcing contributions: (i) a curated dataset of $\sim$9k sketch-to-multiview samples, constructed via an automated generation and filtering pipeline; (ii) Parallel Camera-Aware Attention Adapters (CA3) that inject geometric inductive biases into the video transformer; and (iii) a Sparse Correspondence Supervision Loss (CSL) derived from Structure-from-Motion reconstructions. Our framework synthesizes all views in a single denoising process without requiring reference images, iterative refinement, or per-scene optimization. Our approach significantly outperforms state-of-the-art two-stage baselines, improving realism (FID) by over 60% and geometric consistency (Corr-Acc) by 23%, while providing up to a 3.7$\times$ inference speedup.

Related papers

• CasDeblurGS: Cascaded 2D-to-3D Multi-View Consistency for 3D Gaussian Splatting from Two Blurry Images [2.12833538322985]
  Free-viewpoint 3D scene media is increasingly important for immersive applications, but practical capture often suffers from view sparsity and motion blur.<n>We propose CasDeblurGS, a cascaded framework that progressively recovers reliable cross-view information from local 2D correspondences to global 3D guidance.<n> Experiments on real-world and synthetic Deblur-NeRF scenes show consistent gains over strong baselines, improving PSNR by 1.19 dB and 2.11 dB, respectively.
  arXiv  Detail & Related papers  (2026-08-11T01:13:21Z)
• SCOPE: Scale-Consistent One-Pass Estimation of 3D Geometry [69.89196162649091]
  SCOPE (Scale-Consistent One-Pass Estimation of 3D Geometry) is a novel approach for estimating 3D geometry from extended monocular video sequences.<n>Our approach generates affine-invariant 3D point maps with shared parameters across entire sequences, enabling consistent scale-invariant representations.
  arXiv  Detail & Related papers  (2026-06-19T10:23:25Z)
• IVGT: Implicit Visual Geometry Transformer for Neural Scene Representation [76.36174247570716]
  We propose IVGT, an Implicit Visual Geometry Transformer that implicitly models continuous and coherent geometry from pose-free multi-view images.<n> IVGT learns a continuous neural scene representation in a canonical coordinate system and supports continuous spatial queries at any 3D positions.<n>It allows direct extraction of continuous and coherent surface geometry, enabling rendering of RGB images, depth maps, and surface normal maps from arbitrary viewpoints.
  arXiv  Detail & Related papers  (2026-05-15T17:59:57Z)
• GeoNVS: Geometry Grounded Video Diffusion for Novel View Synthesis [73.97869945427645]
  GeoNVS is a novel-view synthesizer that enhances both geometric fidelity and camera controllability through explicit 3D geometric guidance.<n>GS-Adapter lifts input-view diffusion features into 3D Gaussian representations, and adaptively fuses them with diffusion features to correct geometrically inconsistent representations.<n>Experiments across 9 scenes and 18 settings demonstrate state-of-the-art performance, achieving 11.3% and 14.9% improvements over SEVA and CameraCtrl.
  arXiv  Detail & Related papers  (2026-03-16T08:23:00Z)
• Wonder3D++: Cross-domain Diffusion for High-fidelity 3D Generation from a Single Image [68.55613894952177]
  We introduce textbfWonder3D++, a novel method for efficiently generating high-fidelity textured meshes from single-view images.<n>We propose a cross-domain diffusion model that generates multi-view normal maps and the corresponding color images.<n> Lastly, we introduce a cascaded 3D mesh extraction algorithm that drives high-quality surfaces from the multi-view 2D representations in only about $3$ minute in a coarse-to-fine manner.
  arXiv  Detail & Related papers  (2025-11-03T17:24:18Z)
• GaussVideoDreamer: 3D Scene Generation with Video Diffusion and Inconsistency-Aware Gaussian Splatting [17.17292309504131]
  GaussVideoDreamer advances generative multimedia approaches by bridging the gap between image, video, and 3D generation.<n>Our approach achieves 32% higher LLaVA-IQA scores and at least 2x speedup compared to existing methods.
  arXiv  Detail & Related papers  (2025-04-14T09:04:01Z)
• T$^3$-S2S: Training-free Triplet Tuning for Sketch to Scene Generation [56.054622766743414]
  We propose a Training-free Triplet Tuning for Sketch-to-Scene (T3-S2S) generation.<n>It enhances keyword representation via the prompt balance module, reducing the risk of missing critical instances.<n>Experiments validate that our triplet tuning approach substantially improves the performance of existing sketch-to-image models.
  arXiv  Detail & Related papers  (2024-12-18T04:01:32Z)
• Geometry-Biased Transformer for Robust Multi-View 3D Human Pose Reconstruction [3.069335774032178]
  We propose a novel encoder-decoder Transformer architecture to estimate 3D poses from multi-view 2D pose sequences. We conduct experiments on three benchmark public datasets, Human3.6M, CMU Panoptic and Occlusion-Persons.
  arXiv  Detail & Related papers  (2023-12-28T16:30:05Z)
• Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models [16.326276673056334]
  Consistent-1-to-3 is a generative framework that significantly mitigates this issue. We decompose the NVS task into two stages: (i) transforming observed regions to a novel view, and (ii) hallucinating unseen regions. We propose to employ epipolor-guided attention to incorporate geometry constraints, and multi-view attention to better aggregate multi-view information.
  arXiv  Detail & Related papers  (2023-10-04T17:58:57Z)
• Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis [48.33860286920389]
  3D-aware image synthesis aims to generate images of objects from multiple views by learning a 3D representation. Existing approaches lack geometry constraints, hence usually fail to generate multi-view consistent images. We propose Multi-View Consistent Generative Adrial Networks (MVCGAN) for high-quality 3D-aware image synthesis with geometry constraints.
  arXiv  Detail & Related papers  (2022-04-13T11:23:09Z)
• Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo [71.59494156155309]
  Existing approaches for multi-view 3D pose estimation explicitly establish cross-view correspondences to group 2D pose detections from multiple camera views. We present our multi-view 3D pose estimation approach based on plane sweep stereo to jointly address the cross-view fusion and 3D pose reconstruction in a single shot.
  arXiv  Detail & Related papers  (2021-04-06T03:49:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.