X-Ray: A Sequential 3D Representation For Generation

• URL: http://arxiv.org/abs/2404.14329v2
• Date: Sun, 2 Jun 2024 01:58:41 GMT
• Title: X-Ray: A Sequential 3D Representation For Generation
• Authors: Tao Hu, Wenhang Ge, Yuyang Zhao, Gim Hee Lee,
• Abstract summary: We introduce X-Ray, a novel 3D sequential representation inspired by x-ray scans. X-Ray transforms a 3D object into a series of surface frames at different layers, making it suitable for generating 3D models from images.
• Score: 54.160173837582796
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce X-Ray, a novel 3D sequential representation inspired by the penetrability of x-ray scans. X-Ray transforms a 3D object into a series of surface frames at different layers, making it suitable for generating 3D models from images. Our method utilizes ray casting from the camera center to capture geometric and textured details, including depth, normal, and color, across all intersected surfaces. This process efficiently condenses the whole 3D object into a multi-frame video format, motivating the utilize of a network architecture similar to those in video diffusion models. This design ensures an efficient 3D representation by focusing solely on surface information. Also, we propose a two-stage pipeline to generate 3D objects from X-Ray Diffusion Model and Upsampler. We demonstrate the practicality and adaptability of our X-Ray representation by synthesizing the complete visible and hidden surfaces of a 3D object from a single input image. Experimental results reveal the state-of-the-art superiority of our representation in enhancing the accuracy of 3D generation, paving the way for new 3D representation research and practical applications.

Related papers

• Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priors [17.544733016978928]
  3D object generation from a single image involves estimating the full 3D geometry and texture of unseen views from an unposed RGB image captured in the wild. Recent advancements in 3D object generation have introduced techniques that reconstruct an object's 3D shape and texture. We propose bridging the gap between 2D and 3D diffusion models to address this limitation.
  arXiv  Detail & Related papers  (2024-10-12T10:14:11Z)
• Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text [61.9973218744157]
  We introduce Director3D, a robust open-world text-to-3D generation framework, designed to generate both real-world 3D scenes and adaptive camera trajectories. Experiments demonstrate that Director3D outperforms existing methods, offering superior performance in real-world 3D generation.
  arXiv  Detail & Related papers  (2024-06-25T14:42:51Z)
• DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data [50.164670363633704]
  We present DIRECT-3D, a diffusion-based 3D generative model for creating high-quality 3D assets from text prompts. Our model is directly trained on extensive noisy and unaligned in-the-wild' 3D assets. We achieve state-of-the-art performance in both single-class generation and text-to-3D generation.
  arXiv  Detail & Related papers  (2024-06-06T17:58:15Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation [68.06991943974195]
  We present RenderDiffusion, the first diffusion model for 3D generation and inference, trained using only monocular 2D supervision. We evaluate RenderDiffusion on FFHQ, AFHQ, ShapeNet and CLEVR datasets, showing competitive performance for generation of 3D scenes and inference of 3D scenes from 2D images.
  arXiv  Detail & Related papers  (2022-11-17T20:17:04Z)
• XDGAN: Multi-Modal 3D Shape Generation in 2D Space [60.46777591995821]
  We propose a novel method to convert 3D shapes into compact 1-channel geometry images and leverage StyleGAN3 and image-to-image translation networks to generate 3D objects in 2D space. The generated geometry images are quick to convert to 3D meshes, enabling real-time 3D object synthesis, visualization and interactive editing. We show both quantitatively and qualitatively that our method is highly effective at various tasks such as 3D shape generation, single view reconstruction and shape manipulation, while being significantly faster and more flexible compared to recent 3D generative models.
  arXiv  Detail & Related papers  (2022-10-06T15:54:01Z)
• 3D-GIF: 3D-Controllable Object Generation via Implicit Factorized Representations [31.095503715696722]
  We propose the factorized representations which are view-independent and light-disentangled, and training schemes with randomly sampled light conditions. We demonstrate the superiority of our method by visualizing factorized representations, re-lighted images, and albedo-textured meshes. This is the first work that extracts albedo-textured meshes with unposed 2D images without any additional labels or assumptions.
  arXiv  Detail & Related papers  (2022-03-12T15:23:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.