Textured 3D Regenerative Morphing with 3D Diffusion Prior

• URL: http://arxiv.org/abs/2502.14316v1
• Date: Thu, 20 Feb 2025 07:02:22 GMT
• Title: Textured 3D Regenerative Morphing with 3D Diffusion Prior
• Authors: Songlin Yang, Yushi Lan, Honghua Chen, Xingang Pan,
• Abstract summary: Textured 3D morphing creates smooth and plausible sequences between two 3D objects. Previous methods rely on establishing point-to-point correspondences and determining smooth deformation trajectories. We propose a method for 3D regenerative morphing using a 3D diffusion prior.
• Score: 29.7508625572437
• License:
• Abstract: Textured 3D morphing creates smooth and plausible interpolation sequences between two 3D objects, focusing on transitions in both shape and texture. This is important for creative applications like visual effects in filmmaking. Previous methods rely on establishing point-to-point correspondences and determining smooth deformation trajectories, which inherently restrict them to shape-only morphing on untextured, topologically aligned datasets. This restriction leads to labor-intensive preprocessing and poor generalization. To overcome these challenges, we propose a method for 3D regenerative morphing using a 3D diffusion prior. Unlike previous methods that depend on explicit correspondences and deformations, our method eliminates the additional need for obtaining correspondence and uses the 3D diffusion prior to generate morphing. Specifically, we introduce a 3D diffusion model and interpolate the source and target information at three levels: initial noise, model parameters, and condition features. We then explore an Attention Fusion strategy to generate more smooth morphing sequences. To further improve the plausibility of semantic interpolation and the generated 3D surfaces, we propose two strategies: (a) Token Reordering, where we match approximate tokens based on semantic analysis to guide implicit correspondences in the denoising process of the diffusion model, and (b) Low-Frequency Enhancement, where we enhance low-frequency signals in the tokens to improve the quality of generated surfaces. Experimental results show that our method achieves superior smoothness and plausibility in 3D morphing across diverse cross-category object pairs, offering a novel regenerative method for 3D morphing with textured representations.

Related papers

• Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priors [17.544733016978928]
  3D object generation from a single image involves estimating the full 3D geometry and texture of unseen views from an unposed RGB image captured in the wild. Recent advancements in 3D object generation have introduced techniques that reconstruct an object's 3D shape and texture. We propose bridging the gap between 2D and 3D diffusion models to address this limitation.
  arXiv  Detail & Related papers  (2024-10-12T10:14:11Z)
• DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation [10.250715657201363]
  We introduce DreamMesh4D, a novel framework combining mesh representation with geometric skinning technique to generate high-quality 4D object from a monocular video. Our method is compatible with modern graphic pipelines, showcasing its potential in the 3D gaming and film industry.
  arXiv  Detail & Related papers  (2024-10-09T10:41:08Z)
• NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows [60.291277312569285]
  We present a method for automatically modifying a NeRF representation based on a single observation. Our method defines the transformation as a 3D flow, specifically as a weighted linear blending of rigid transformations. We also introduce a new dataset for exploring the problem of modifying a NeRF scene through a single observation.
  arXiv  Detail & Related papers  (2024-06-15T07:58:08Z)
• PolyDiff: Generating 3D Polygonal Meshes with Diffusion Models [15.846449180313778]
  PolyDiff is the first diffusion-based approach capable of directly generating realistic and diverse 3D polygonal meshes. Our model is capable of producing high-quality 3D polygonal meshes, ready for integration into downstream 3D.
  arXiv  Detail & Related papers  (2023-12-18T18:19:26Z)
• SE(3) Diffusion Model-based Point Cloud Registration for Robust 6D Object Pose Estimation [66.16525145765604]
  We introduce an SE(3) diffusion model-based point cloud registration framework for 6D object pose estimation in real-world scenarios. Our approach formulates the 3D registration task as a denoising diffusion process, which progressively refines the pose of the source point cloud. Experiments demonstrate that our diffusion registration framework presents outstanding pose estimation performance on the real-world TUD-L, LINEMOD, and Occluded-LINEMOD datasets.
  arXiv  Detail & Related papers  (2023-10-26T12:47:26Z)
• ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections [71.46546520120162]
  Estimating 3D articulated shapes like animal bodies from monocular images is inherently challenging. We propose ARTIC3D, a self-supervised framework to reconstruct per-instance 3D shapes from a sparse image collection in-the-wild. We produce realistic animations by fine-tuning the rendered shape and texture under rigid part transformations.
  arXiv  Detail & Related papers  (2023-06-07T17:47:50Z)
• MoDA: Modeling Deformable 3D Objects from Casual Videos [84.29654142118018]
  We propose neural dual quaternion blend skinning (NeuDBS) to achieve 3D point deformation without skin-collapsing artifacts. In the endeavor to register 2D pixels across different frames, we establish a correspondence between canonical feature embeddings that encodes 3D points within the canonical space. Our approach can reconstruct 3D models for humans and animals with better qualitative and quantitative performance than state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-04-17T13:49:04Z)
• Vox-E: Text-guided Voxel Editing of 3D Objects [14.88446525549421]
  Large scale text-guided diffusion models have garnered significant attention due to their ability to synthesize diverse images. We present a technique that harnesses the power of latent diffusion models for editing existing 3D objects.
  arXiv  Detail & Related papers  (2023-03-21T17:36:36Z)
• 3D Neural Field Generation using Triplane Diffusion [37.46688195622667]
  We present an efficient diffusion-based model for 3D-aware generation of neural fields. Our approach pre-processes training data, such as ShapeNet meshes, by converting them to continuous occupancy fields. We demonstrate state-of-the-art results on 3D generation on several object classes from ShapeNet.
  arXiv  Detail & Related papers  (2022-11-30T01:55:52Z)
• DreamFusion: Text-to-3D using 2D Diffusion [52.52529213936283]
  Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.
  arXiv  Detail & Related papers  (2022-09-29T17:50:40Z)
• Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild [96.09941542587865]
  We present a novel 3D pose refinement approach based on differentiable rendering for objects of arbitrary categories in the wild. In this way, we precisely align 3D models to objects in RGB images which results in significantly improved 3D pose estimates. We evaluate our approach on the challenging Pix3D dataset and achieve up to 55% relative improvement compared to state-of-the-art refinement methods in multiple metrics.
  arXiv  Detail & Related papers  (2020-07-17T12:34:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.