GenHSI: Controllable Generation of Human-Scene Interaction Videos

• URL: http://arxiv.org/abs/2506.19840v1
• Date: Tue, 24 Jun 2025 17:58:04 GMT
• Title: GenHSI: Controllable Generation of Human-Scene Interaction Videos
• Authors: Zekun Li, Rui Zhou, Rahul Sajnani, Xiaoyan Cong, Daniel Ritchie, Srinath Sridhar,
• Abstract summary: GenHSI is a training-free method for controllable generation of long human-scene interaction videos.<n>Taking inspiration from movie animation, our key insight is to overcome the limitations of previous work by subdividing the long video generation task into three stages.<n>We are the first to generate a long video sequence with a consistent camera pose that contains arbitrary numbers of character actions without training.
• Score: 22.186091372007105
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale pre-trained video diffusion models have exhibited remarkable capabilities in diverse video generation. However, existing solutions face several challenges in using these models to generate long movie-like videos with rich human-object interactions that include unrealistic human-scene interaction, lack of subject identity preservation, and require expensive training. We propose GenHSI, a training-free method for controllable generation of long human-scene interaction videos (HSI). Taking inspiration from movie animation, our key insight is to overcome the limitations of previous work by subdividing the long video generation task into three stages: (1) script writing, (2) pre-visualization, and (3) animation. Given an image of a scene, a user description, and multiple images of a person, we use these three stages to generate long-videos that preserve human-identity and provide rich human-scene interactions. Script writing converts complex human tasks into simple atomic tasks that are used in the pre-visualization stage to generate 3D keyframes (storyboards). These 3D keyframes are rendered and animated by off-the-shelf video diffusion models for consistent long video generation with rich contacts in a 3D-aware manner. A key advantage of our work is that we alleviate the need for scanned, accurate scenes and create 3D keyframes from single-view images. We are the first to generate a long video sequence with a consistent camera pose that contains arbitrary numbers of character actions without training. Experiments demonstrate that our method can generate long videos that effectively preserve scene content and character identity with plausible human-scene interaction from a single image scene. Visit our project homepage https://kunkun0w0.github.io/project/GenHSI/ for more information.

Related papers

• Video Perception Models for 3D Scene Synthesis [109.5543506037003]
  VIPScene is a novel framework that exploits the encoded commonsense knowledge of the 3D physical world in video generation models.<n>VIPScene seamlessly integrates video generation, feedforward 3D reconstruction, and open-vocabulary perception models to semantically and geometrically analyze each object in a scene.
  arXiv  Detail & Related papers  (2025-06-25T16:40:17Z)
• Interspatial Attention for Efficient 4D Human Video Generation [98.36274427702915]
  We introduce a new interspatial attention (ISA) mechanism as a scalable building block for modern video generation models.<n>ISA is a new type of cross attention that uses relative positional encodings tailored for the generation of human videos.<n>Our model achieves state-of-the-art performance for 4D human video synthesis, demonstrating remarkable motion consistency and identity preservation.
  arXiv  Detail & Related papers  (2025-05-21T17:53:47Z)
• Multi-identity Human Image Animation with Structural Video Diffusion [64.20452431561436]
  We present Structural Video Diffusion, a novel framework for generating realistic multi-human videos.<n>Our approach introduces identity-specific embeddings to maintain consistent appearances across individuals.<n>We expand existing human video dataset with 25K new videos featuring diverse multi-human and object interaction scenarios.
  arXiv  Detail & Related papers  (2025-04-05T10:03:49Z)
• VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual intervention [70.61101071902596]
  Current video generation models excel at short clips but fail to produce cohesive multi-shot narratives due to disjointed visual dynamics and fractured storylines.<n>We introduce VideoGen-of-Thought (VGoT), a step-by-step framework that automates multi-shot video synthesis from a single sentence.<n>VGoT generates multi-shot videos that outperform state-of-the-art baselines by 20.4% in within-shot face consistency and 17.4% in style consistency.
  arXiv  Detail & Related papers  (2024-12-03T08:33:50Z)
• Gaussians-to-Life: Text-Driven Animation of 3D Gaussian Splatting Scenes [49.26872036160368]
  We propose a method for animating parts of high-quality 3D scenes in a Gaussian Splatting representation.<n>We find that, in contrast to prior work, this enables realistic animations of complex, pre-existing 3D scenes.
  arXiv  Detail & Related papers  (2024-11-28T16:01:58Z)
• HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation [64.37874983401221]
  We present HumanVid, the first large-scale high-quality dataset tailored for human image animation. For the real-world data, we compile a vast collection of real-world videos from the internet. For the synthetic data, we collected 10K 3D avatar assets and leveraged existing assets of body shapes, skin textures and clothings.
  arXiv  Detail & Related papers  (2024-07-24T17:15:58Z)
• MultiPly: Reconstruction of Multiple People from Monocular Video in the Wild [32.6521941706907]
  We present MultiPly, a novel framework to reconstruct multiple people in 3D from monocular in-the-wild videos. We first define a layered neural representation for the entire scene, composited by individual human and background models. We learn the layered neural representation from videos via our layer-wise differentiable volume rendering.
  arXiv  Detail & Related papers  (2024-06-03T17:59:57Z)
• GenZI: Zero-Shot 3D Human-Scene Interaction Generation [39.9039943099911]
  We propose GenZI, the first zero-shot approach to generating 3D human-scene interactions. Key to GenZI is our distillation of interaction priors from large vision-language models (VLMs), which have learned a rich semantic space of 2D human-scene compositions. In contrast to existing learning-based approaches, GenZI circumvents the conventional need for captured 3D interaction data.
  arXiv  Detail & Related papers  (2023-11-29T15:40:11Z)
• Image Comes Dancing with Collaborative Parsing-Flow Video Synthesis [124.48519390371636]
  Transfering human motion from a source to a target person poses great potential in computer vision and graphics applications. Previous work has either relied on crafted 3D human models or trained a separate model specifically for each target person. This work studies a more general setting, in which we aim to learn a single model to parsimoniously transfer motion from a source video to any target person.
  arXiv  Detail & Related papers  (2021-10-27T03:42:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.