SyncDreamer: Generating Multiview-consistent Images from a Single-view Image

• URL: http://arxiv.org/abs/2309.03453v2
• Date: Mon, 15 Apr 2024 10:28:44 GMT
• Title: SyncDreamer: Generating Multiview-consistent Images from a Single-view Image
• Authors: Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, Wenping Wang,
• Abstract summary: A novel diffusion model called SyncDreamer generates multiview-consistent images from a single-view image. Experiments show that SyncDreamer generates images with high consistency across different views.
• Score: 59.75474518708409
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we present a novel diffusion model called that generates multiview-consistent images from a single-view image. Using pretrained large-scale 2D diffusion models, recent work Zero123 demonstrates the ability to generate plausible novel views from a single-view image of an object. However, maintaining consistency in geometry and colors for the generated images remains a challenge. To address this issue, we propose a synchronized multiview diffusion model that models the joint probability distribution of multiview images, enabling the generation of multiview-consistent images in a single reverse process. SyncDreamer synchronizes the intermediate states of all the generated images at every step of the reverse process through a 3D-aware feature attention mechanism that correlates the corresponding features across different views. Experiments show that SyncDreamer generates images with high consistency across different views, thus making it well-suited for various 3D generation tasks such as novel-view-synthesis, text-to-3D, and image-to-3D.

Related papers

• VistaDream: Sampling multiview consistent images for single-view scene reconstruction [63.991582576387856]
  VistaDream is a framework to reconstruct a 3D scene from a single-view image. Recent diffusion models enable generating high-quality novel-view images from a single-view input image.
  arXiv  Detail & Related papers  (2024-10-22T10:55:59Z)
• MultiDiff: Consistent Novel View Synthesis from a Single Image [60.04215655745264]
  MultiDiff is a novel approach for consistent novel view synthesis of scenes from a single RGB image. Our results demonstrate that MultiDiff outperforms state-of-the-art methods on the challenging, real-world datasets RealEstate10K and ScanNet.
  arXiv  Detail & Related papers  (2024-06-26T17:53:51Z)
• Bootstrap3D: Improving Multi-view Diffusion Model with Synthetic Data [80.92268916571712]
  A critical bottleneck is the scarcity of high-quality 3D objects with detailed captions. We propose Bootstrap3D, a novel framework that automatically generates an arbitrary quantity of multi-view images. We have generated 1 million high-quality synthetic multi-view images with dense descriptive captions.
  arXiv  Detail & Related papers  (2024-05-31T17:59:56Z)
• MVDiff: Scalable and Flexible Multi-View Diffusion for 3D Object Reconstruction from Single-View [0.0]
  This paper proposes a general framework to generate consistent multi-view images from single image or leveraging scene representation transformer and view-conditioned diffusion model. Our model is able to generate 3D meshes surpassing baselines methods in evaluation metrics, including PSNR, SSIM and LPIPS.
  arXiv  Detail & Related papers  (2024-05-06T22:55:53Z)
• DreamComposer: Controllable 3D Object Generation via Multi-View Conditions [45.4321454586475]
  Recent works are capable of generating high-quality novel views from a single in-the-wild image. Due to the lack of information from multiple views, these works encounter difficulties in generating controllable novel views. We present DreamComposer, a flexible and scalable framework that can enhance existing view-aware diffusion models by injecting multi-view conditions.
  arXiv  Detail & Related papers  (2023-12-06T16:55:53Z)
• ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion [61.37481051263816]
  Given a single image of a 3D object, this paper proposes a method (named ConsistNet) that is able to generate multiple images of the same object. Our method effectively learns 3D consistency over a frozen Zero123 backbone and can generate 16 surrounding views of the object within 40 seconds on a single A100 GPU.
  arXiv  Detail & Related papers  (2023-10-16T12:29:29Z)
• Consistent123: Improve Consistency for One Image to 3D Object Synthesis [74.1094516222327]
  Large image diffusion models enable novel view synthesis with high quality and excellent zero-shot capability. These models have no guarantee of view consistency, limiting the performance for downstream tasks like 3D reconstruction and image-to-3D generation. We propose Consistent123 to synthesize novel views simultaneously by incorporating additional cross-view attention layers and the shared self-attention mechanism.
  arXiv  Detail & Related papers  (2023-10-12T07:38:28Z)
• Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis [48.33860286920389]
  3D-aware image synthesis aims to generate images of objects from multiple views by learning a 3D representation. Existing approaches lack geometry constraints, hence usually fail to generate multi-view consistent images. We propose Multi-View Consistent Generative Adrial Networks (MVCGAN) for high-quality 3D-aware image synthesis with geometry constraints.
  arXiv  Detail & Related papers  (2022-04-13T11:23:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.