MVPaint: Synchronized Multi-View Diffusion for Painting Anything 3D

• URL: http://arxiv.org/abs/2411.02336v1
• Date: Mon, 04 Nov 2024 17:59:39 GMT
• Title: MVPaint: Synchronized Multi-View Diffusion for Painting Anything 3D
• Authors: Wei Cheng, Juncheng Mu, Xianfang Zeng, Xin Chen, Anqi Pang, Chi Zhang, Zhibin Wang, Bin Fu, Gang Yu, Ziwei Liu, Liang Pan,
• Abstract summary: Texturing is a 3D asset production, which enhances the visual appeal and visual appeal. Despite recent advancements, methods often yield subpar results, primarily due to local discontinuities. We propose a novel framework called MVPaint, which can generate high-resolution, seamless multiview consistency.
• Score: 63.9188712646076
• License:
• Abstract: Texturing is a crucial step in the 3D asset production workflow, which enhances the visual appeal and diversity of 3D assets. Despite recent advancements in Text-to-Texture (T2T) generation, existing methods often yield subpar results, primarily due to local discontinuities, inconsistencies across multiple views, and their heavy dependence on UV unwrapping outcomes. To tackle these challenges, we propose a novel generation-refinement 3D texturing framework called MVPaint, which can generate high-resolution, seamless textures while emphasizing multi-view consistency. MVPaint mainly consists of three key modules. 1) Synchronized Multi-view Generation (SMG). Given a 3D mesh model, MVPaint first simultaneously generates multi-view images by employing an SMG model, which leads to coarse texturing results with unpainted parts due to missing observations. 2) Spatial-aware 3D Inpainting (S3I). To ensure complete 3D texturing, we introduce the S3I method, specifically designed to effectively texture previously unobserved areas. 3) UV Refinement (UVR). Furthermore, MVPaint employs a UVR module to improve the texture quality in the UV space, which first performs a UV-space Super-Resolution, followed by a Spatial-aware Seam-Smoothing algorithm for revising spatial texturing discontinuities caused by UV unwrapping. Moreover, we establish two T2T evaluation benchmarks: the Objaverse T2T benchmark and the GSO T2T benchmark, based on selected high-quality 3D meshes from the Objaverse dataset and the entire GSO dataset, respectively. Extensive experimental results demonstrate that MVPaint surpasses existing state-of-the-art methods. Notably, MVPaint could generate high-fidelity textures with minimal Janus issues and highly enhanced cross-view consistency.

Related papers

• TEXGen: a Generative Diffusion Model for Mesh Textures [63.43159148394021]
  We focus on the fundamental problem of learning in the UV texture space itself. We propose a scalable network architecture that interleaves convolutions on UV maps with attention layers on point clouds. We train a 700 million parameter diffusion model that can generate UV texture maps guided by text prompts and single-view images.
  arXiv  Detail & Related papers  (2024-11-22T05:22:11Z)
• MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing [90.30646271720919]
  Novel View Synthesis (NVS) and 3D generation have recently achieved prominent improvements. We propose MVInpainter, re-formulating the 3D editing as a multi-view 2D inpainting task. MVInpainter partially inpaints multi-view images with the reference guidance rather than intractably generating an entirely novel view from scratch.
  arXiv  Detail & Related papers  (2024-08-15T07:57:28Z)
• VCD-Texture: Variance Alignment based 3D-2D Co-Denoising for Text-Guided Texturing [22.39760469467524]
  We propose a Variance texture synthesis to address the modal gap between the 2D and 3D diffusion models. We present an inpainting module to improve details with conflicting regions.
  arXiv  Detail & Related papers  (2024-07-05T12:11:33Z)
• TexDreamer: Towards Zero-Shot High-Fidelity 3D Human Texture Generation [41.959089177835764]
  TexDreamer is the first zero-shot multimodal high-fidelity 3D human texture generation model. We introduce ArTicuLated humAn textureS, the largest high-resolution (1024 X 1024) 3D human texture dataset.
  arXiv  Detail & Related papers  (2024-03-19T17:02:07Z)
• Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models [34.715076045118444]
  Paint3D produces high-resolution, lighting-less, and diverse 2K UV texture maps for un-textured 3D meshes on text or image inputs.
  arXiv  Detail & Related papers  (2023-12-21T15:01:47Z)
• Repaint123: Fast and High-quality One Image to 3D Generation with Progressive Controllable 2D Repainting [16.957766297050707]
  We present Repaint123 to alleviate multi-view bias as well as texture degradation and speed up the generation process. We propose visibility-aware adaptive repainting strength for overlap regions to enhance the generated image quality. Our method has a superior ability to generate high-quality 3D content with multi-view consistency and fine textures in 2 minutes from scratch.
  arXiv  Detail & Related papers  (2023-12-20T18:51:02Z)
• TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models [77.85129451435704]
  We present a new method to synthesize textures for 3D, using large-scale-guided image diffusion models. Specifically, we leverage latent diffusion models, apply the set denoising model and aggregate denoising text map.
  arXiv  Detail & Related papers  (2023-10-20T19:15:29Z)
• AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis [78.17671694498185]
  We propose AUV-Net which learns to embed 3D surfaces into a 2D aligned UV space. As a result, textures are aligned across objects, and can thus be easily synthesized by generative models of images. The learned UV mapping and aligned texture representations enable a variety of applications including texture transfer, texture synthesis, and textured single view 3D reconstruction.
  arXiv  Detail & Related papers  (2022-04-06T21:39:24Z)
• OSTeC: One-Shot Texture Completion [86.23018402732748]
  We propose an unsupervised approach for one-shot 3D facial texture completion. The proposed approach rotates an input image in 3D and fill-in the unseen regions by reconstructing the rotated image in a 2D face generator. We frontalize the target image by projecting the completed texture into the generator.
  arXiv  Detail & Related papers  (2020-12-30T23:53:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.