Bringing Objects to Life: 4D generation from 3D objects

• URL: http://arxiv.org/abs/2412.20422v1
• Date: Sun, 29 Dec 2024 10:12:01 GMT
• Title: Bringing Objects to Life: 4D generation from 3D objects
• Authors: Ohad Rahamim, Ori Malca, Dvir Samuel, Gal Chechik,
• Abstract summary: We introduce a method for animating user-provided 3D objects by conditioning on textual prompts to guide 4D generation. Our method achieves up to threefold improvements in identity preservation measured using LPIPS scores.
• Score: 31.533802484121182
• License:
• Abstract: Recent advancements in generative modeling now enable the creation of 4D content (moving 3D objects) controlled with text prompts. 4D generation has large potential in applications like virtual worlds, media, and gaming, but existing methods provide limited control over the appearance and geometry of generated content. In this work, we introduce a method for animating user-provided 3D objects by conditioning on textual prompts to guide 4D generation, enabling custom animations while maintaining the identity of the original object. We first convert a 3D mesh into a ``static" 4D Neural Radiance Field (NeRF) that preserves the visual attributes of the input object. Then, we animate the object using an Image-to-Video diffusion model driven by text. To improve motion realism, we introduce an incremental viewpoint selection protocol for sampling perspectives to promote lifelike movement and a masked Score Distillation Sampling (SDS) loss, which leverages attention maps to focus optimization on relevant regions. We evaluate our model in terms of temporal coherence, prompt adherence, and visual fidelity and find that our method outperforms baselines that are based on other approaches, achieving up to threefold improvements in identity preservation measured using LPIPS scores, and effectively balancing visual quality with dynamic content.

Related papers

• 3D Object Manipulation in a Single Image using Generative Models [30.241857090353864]
  We introduce textbfOMG3D, a novel framework that integrates the precise geometric control with the generative power of diffusion models. Our framework first converts 2D objects into 3D, enabling user-directed modifications and lifelike motions at the geometric level. Remarkably, all these steps can be done using one NVIDIA 3090.
  arXiv  Detail & Related papers  (2025-01-22T15:06:30Z)
• AnimateMe: 4D Facial Expressions via Diffusion Models [72.63383191654357]
  Recent advances in diffusion models have enhanced the capabilities of generative models in 2D animation. We employ Graph Neural Networks (GNNs) as denoising diffusion models in a novel approach, formulating the diffusion process directly on the mesh space. This facilitates the generation of facial deformations through a mesh-diffusion-based model.
  arXiv  Detail & Related papers  (2024-03-25T21:40:44Z)
• Comp4D: LLM-Guided Compositional 4D Scene Generation [65.5810466788355]
  We present Comp4D, a novel framework for Compositional 4D Generation. Unlike conventional methods that generate a singular 4D representation of the entire scene, Comp4D innovatively constructs each 4D object within the scene separately. Our method employs a compositional score distillation technique guided by the pre-defined trajectories.
  arXiv  Detail & Related papers  (2024-03-25T17:55:52Z)
• ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models [65.22994156658918]
  We present a method that learns to generate multi-view images in a single denoising process from real-world data. We design an autoregressive generation that renders more 3D-consistent images at any viewpoint.
  arXiv  Detail & Related papers  (2024-03-04T07:57:05Z)
• 4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency [118.15258850780417]
  We present textbf4DGen, a novel framework for grounded 4D content creation. Our pipeline facilitates controllable 4D generation, enabling users to specify the motion via monocular video or adopt image-to-video generations. Compared to existing video-to-4D baselines, our approach yields superior results in faithfully reconstructing input signals.
  arXiv  Detail & Related papers  (2023-12-28T18:53:39Z)
• A Unified Approach for Text- and Image-guided 4D Scene Generation [58.658768832653834]
  We propose Dream-in-4D, which features a novel two-stage approach for text-to-4D synthesis. We show that our approach significantly advances image and motion quality, 3D consistency and text fidelity for text-to-4D generation. Our method offers, for the first time, a unified approach for text-to-4D, image-to-4D and personalized 4D generation tasks.
  arXiv  Detail & Related papers  (2023-11-28T15:03:53Z)
• Text-To-4D Dynamic Scene Generation [111.89517759596345]
  We present MAV3D (Make-A-Video3D), a method for generating three-dimensional dynamic scenes from text descriptions. Our approach uses a 4D dynamic Neural Radiance Field (NeRF), which is optimized for scene appearance, density, and motion consistency. The dynamic video output generated from the provided text can be viewed from any camera location and angle, and can be composited into any 3D environment.
  arXiv  Detail & Related papers  (2023-01-26T18:14:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.