WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions

• URL: http://arxiv.org/abs/2505.18151v1
• Date: Fri, 23 May 2025 17:59:24 GMT
• Title: WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions
• Authors: Zizhang Li, Hong-Xing Yu, Wei Liu, Yin Yang, Charles Herrmann, Gordon Wetzstein, Jiajun Wu,
• Abstract summary: WonderPlay is a framework integrating physics simulation and video generation.<n>It generates action-conditioned dynamic 3D scenes from a single image.<n>WonderPlay enables users to interact with various scenes of diverse content.
• Score: 49.43000450846916
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: WonderPlay is a novel framework integrating physics simulation with video generation for generating action-conditioned dynamic 3D scenes from a single image. While prior works are restricted to rigid body or simple elastic dynamics, WonderPlay features a hybrid generative simulator to synthesize a wide range of 3D dynamics. The hybrid generative simulator first uses a physics solver to simulate coarse 3D dynamics, which subsequently conditions a video generator to produce a video with finer, more realistic motion. The generated video is then used to update the simulated dynamic 3D scene, closing the loop between the physics solver and the video generator. This approach enables intuitive user control to be combined with the accurate dynamics of physics-based simulators and the expressivity of diffusion-based video generators. Experimental results demonstrate that WonderPlay enables users to interact with various scenes of diverse content, including cloth, sand, snow, liquid, smoke, elastic, and rigid bodies -- all using a single image input. Code will be made public. Project website: https://kyleleey.github.io/WonderPlay/

Related papers

• RealWonder: Real-Time Physical Action-Conditioned Video Generation [31.747349682347167]
  We present RealWonder, the first real-time system for action-conditioned video generation from a single image.<n>RealWonder integrates 3D reconstruction from single images, physics simulation, and a distilled video generator requiring only 4 diffusion steps.<n>Our system achieves 13.2 FPS at 480x832 resolution, enabling interactive exploration of forces, robot actions, and camera controls on rigid objects.
  arXiv  Detail & Related papers  (2026-03-05T18:22:54Z)
• MotionPhysics: Learnable Motion Distillation for Text-Guided Simulation [25.78198969054392]
  MotionPhysics is an end-to-end differentiable framework that infers plausible physical parameters from a user-provided natural language prompt.<n>We evaluate MotionPhysics across more than thirty scenarios, including real-world, human-designed, and AI-generated 3D objects.
  arXiv  Detail & Related papers  (2026-01-01T22:56:37Z)
• Seed3D 1.0: From Images to High-Fidelity Simulation-Ready 3D Assets [63.67760219308476]
  We present Seed3D 1.0, a foundation model that generates simulation-ready 3D assets from single images.<n>Unlike existing 3D generation models, our system produces assets with accurate geometry, well-aligned textures, and realistic physically-based materials.
  arXiv  Detail & Related papers  (2025-10-22T18:16:32Z)
• ControlHair: Physically-based Video Diffusion for Controllable Dynamic Hair Rendering [13.193640874747745]
  We present ControlHair, a hybrid framework that integrates a physics simulator with conditional video diffusion to enable controllable dynamic hair rendering.<n>ControlHair adopts a three-stage pipeline: it first encodes physics parameters into per-frame geometry using a simulator, then extracts per-frame control signals, and finally feeds control signals into a video diffusion model to generate videos with desired hair dynamics.<n> trained on a 10K video dataset, ControlHair outperforms text- and pose-conditioned baselines, delivering precisely controlled hair dynamics.
  arXiv  Detail & Related papers  (2025-09-25T20:29:05Z)
• Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation [87.91642226587294]
  Current learning-based 3D reconstruction methods rely on the availability of captured real-world multi-view data.<n>We propose a self-distillation framework that distills the implicit 3D knowledge in the video diffusion models into an explicit 3D Gaussian Splatting (3DGS) representation.<n>Our framework achieves state-of-the-art performance in static and dynamic 3D scene generation.
  arXiv  Detail & Related papers  (2025-09-23T17:58:01Z)
• Learning 3D-Gaussian Simulators from RGB Videos [20.250137125726265]
  3DGSim is a 3D physics simulator that learns object dynamics end-to-end from multi-view RGB videos.<n>It encodes images into a 3D Gaussian particle representation, propagates dynamics via a transformer, and renders frames using 3D Gaussian splatting.
  arXiv  Detail & Related papers  (2025-03-31T12:33:59Z)
• Animating the Uncaptured: Humanoid Mesh Animation with Video Diffusion Models [71.78723353724493]
  Animation of humanoid characters is essential in various graphics applications.<n>We propose an approach to synthesize 4D animated sequences of input static 3D humanoid meshes.
  arXiv  Detail & Related papers  (2025-03-20T10:00:22Z)
• Gaussians-to-Life: Text-Driven Animation of 3D Gaussian Splatting Scenes [49.26872036160368]
  We propose a method for animating parts of high-quality 3D scenes in a Gaussian Splatting representation.<n>We find that, in contrast to prior work, this enables realistic animations of complex, pre-existing 3D scenes.
  arXiv  Detail & Related papers  (2024-11-28T16:01:58Z)
• PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation [29.831214435147583]
  We present PhysGen, a novel image-to-video generation method. It produces a realistic, physically plausible, and temporally consistent video. Our key insight is to integrate model-based physical simulation with a data-driven video generation process.
  arXiv  Detail & Related papers  (2024-09-27T17:59:57Z)
• VideoPhy: Evaluating Physical Commonsense for Video Generation [93.28748850301949]
  We present VideoPhy, a benchmark designed to assess whether the generated videos follow physical commonsense for real-world activities. We then generate videos conditioned on captions from diverse state-of-the-art text-to-video generative models. Our human evaluation reveals that the existing models severely lack the ability to generate videos adhering to the given text prompts.
  arXiv  Detail & Related papers  (2024-06-05T17:53:55Z)
• DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors [75.83647027123119]
  We propose to learn the physical properties of a material field with video diffusion priors.<n>We then utilize a physics-based Material-Point-Method simulator to generate 4D content with realistic motions.
  arXiv  Detail & Related papers  (2024-06-03T16:05:25Z)
• Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis [43.02778060969546]
  We propose a controllable monocular dynamic view synthesis pipeline. Our model does not require depth as input, and does not explicitly model 3D scene geometry. We believe our framework can potentially unlock powerful applications in rich dynamic scene understanding, perception for robotics, and interactive 3D video viewing experiences for virtual reality.
  arXiv  Detail & Related papers  (2024-05-23T17:59:52Z)
• Learning 3D Particle-based Simulators from RGB-D Videos [15.683877597215494]
  We propose a method for learning simulators directly from observations. Visual Particle Dynamics (VPD) jointly learns a latent particle-based representation of 3D scenes. Unlike existing 2D video prediction models, VPD's 3D structure enables scene editing and long-term predictions.
  arXiv  Detail & Related papers  (2023-12-08T20:45:34Z)
• Playable Environments: Video Manipulation in Space and Time [98.0621309257937]
  We present Playable Environments - a new representation for interactive video generation and manipulation in space and time. With a single image at inference time, our novel framework allows the user to move objects in 3D while generating a video by providing a sequence of desired actions. Our method builds an environment state for each frame, which can be manipulated by our proposed action module and decoded back to the image space with volumetric rendering.
  arXiv  Detail & Related papers  (2022-03-03T18:51:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.