PanoDreamer: 3D Panorama Synthesis from a Single Image

• URL: http://arxiv.org/abs/2412.04827v1
• Date: Fri, 06 Dec 2024 07:42:48 GMT
• Title: PanoDreamer: 3D Panorama Synthesis from a Single Image
• Authors: Avinash Paliwal, Xilong Zhou, Andrii Tsarov, Nima Khademi Kalantari,
• Abstract summary: PanoDreamer is a novel method for producing a coherent 360$circ$ 3D scene from a single input image.<n>We frame the problem as single-image panorama and depth estimation.
• Score: 7.654478445523882
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we present PanoDreamer, a novel method for producing a coherent 360$^\circ$ 3D scene from a single input image. Unlike existing methods that generate the scene sequentially, we frame the problem as single-image panorama and depth estimation. Once the coherent panoramic image and its corresponding depth are obtained, the scene can be reconstructed by inpainting the small occluded regions and projecting them into 3D space. Our key contribution is formulating single-image panorama and depth estimation as two optimization tasks and introducing alternating minimization strategies to effectively solve their objectives. We demonstrate that our approach outperforms existing techniques in single-image 360$^\circ$ scene reconstruction in terms of consistency and overall quality.

Related papers

• ExScene: Free-View 3D Scene Reconstruction with Gaussian Splatting from a Single Image [4.366356163044466]
  Existing methods are often limited to reconstruct low-consistency 3D scenes with narrow fields of view from single-view input. We propose ExScene, a two-stage pipeline to reconstruct an immersive 3D scene from any given single-view image. ExScene achieves consistent and immersive scene reconstruction using only single-view input.
  arXiv  Detail & Related papers  (2025-03-31T09:33:22Z)
• SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting [53.32467009064287]
  We propose a text-driven 3D-consistent scene generation model: SceneDreamer360. Our proposed method leverages a text-driven panoramic image generation model as a prior for 3D scene generation. Our experiments demonstrate that SceneDreamer360 with its panoramic image generation and 3DGS can produce higher quality, spatially consistent, and visually appealing 3D scenes from any text prompt.
  arXiv  Detail & Related papers  (2024-08-25T02:56:26Z)
• LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation [105.52153675890408]
  3D immersive scene generation is a challenging yet critical task in computer vision and graphics. Layerpano3D is a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt.
  arXiv  Detail & Related papers  (2024-08-23T17:50:23Z)
• Pano2Room: Novel View Synthesis from a Single Indoor Panorama [20.262621556667852]
  Pano2Room is designed to automatically reconstruct high-quality 3D indoor scenes from a single panoramic image. The key idea is to initially construct a preliminary mesh from the input panorama, and iteratively refine this mesh using a panoramic RGBD inpainter. The refined mesh is converted into a 3D Gaussian Splatting field and trained with the collected pseudo novel views.
  arXiv  Detail & Related papers  (2024-08-21T08:19:12Z)
• Sp2360: Sparse-view 360 Scene Reconstruction using Cascaded 2D Diffusion Priors [51.36238367193988]
  We tackle sparse-view reconstruction of a 360 3D scene using priors from latent diffusion models (LDM) We present SparseSplat360, a method that employs a cascade of in-painting and artifact removal models to fill in missing details and clean novel views. Our method generates entire 360 scenes from as few as 9 input views, with a high degree of foreground and background detail.
  arXiv  Detail & Related papers  (2024-05-26T11:01:39Z)
• DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting [56.101576795566324]
  We present a text-to-3D 360$circ$ scene generation pipeline. Our approach utilizes the generative power of a 2D diffusion model and prompt self-refinement. Our method offers a globally consistent 3D scene within a 360$circ$ perspective.
  arXiv  Detail & Related papers  (2024-04-10T10:46:59Z)
• Scaled 360 layouts: Revisiting non-central panoramas [5.2178708158547025]
  We present a novel approach for 3D layout recovery of indoor environments using single non-central panoramas. We exploit the properties of non-central projection systems in a new geometrical processing to recover the scaled layout.
  arXiv  Detail & Related papers  (2024-02-02T14:55:36Z)
• PERF: Panoramic Neural Radiance Field from a Single Panorama [109.31072618058043]
  PERF is a novel view synthesis framework that trains a panoramic neural radiance field from a single panorama. We propose a novel collaborative RGBD inpainting method and a progressive inpainting-and-erasing method to lift up a 360-degree 2D scene to a 3D scene. Our PERF can be widely used for real-world applications, such as panorama-to-3D, text-to-3D, and 3D scene stylization applications.
  arXiv  Detail & Related papers  (2023-10-25T17:59:01Z)
• One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization [30.951405623906258]
  Single image 3D reconstruction is an important but challenging task that requires extensive knowledge of our natural world. We propose a novel method that takes a single image of any object as input and generates a full 360-degree 3D textured mesh in a single feed-forward pass.
  arXiv  Detail & Related papers  (2023-06-29T13:28:16Z)
• High-Resolution Depth Estimation for 360-degree Panoramas through Perspective and Panoramic Depth Images Registration [3.4583104874165804]
  We propose a novel approach to compute high-resolution (2048x1024 and higher) depths for panoramas. Our method generates qualitatively better results than existing panorama-based methods, and further outperforms them quantitatively on datasets unseen by these methods.
  arXiv  Detail & Related papers  (2022-10-19T09:25:12Z)
• DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization [66.25948693095604]
  We propose a novel method for panoramic 3D scene understanding which recovers the 3D room layout and the shape, pose, position, and semantic category for each object from a single full-view panorama image. Experiments demonstrate that our method outperforms existing methods on panoramic scene understanding in terms of both geometry accuracy and object arrangement.
  arXiv  Detail & Related papers  (2021-08-24T13:55:29Z)
• Deep Multi Depth Panoramas for View Synthesis [70.9125433400375]
  We present a novel scene representation - Multi Depth Panorama (MDP) - that consists of multiple RGBD$alpha$ panoramas. MDPs are more compact than previous 3D scene representations and enable high-quality, efficient new view rendering.
  arXiv  Detail & Related papers  (2020-08-04T20:29:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.