FantasyHSI: Video-Generation-Centric 4D Human Synthesis In Any Scene through A Graph-based Multi-Agent Framework

• URL: http://arxiv.org/abs/2509.01232v1
• Date: Mon, 01 Sep 2025 08:20:50 GMT
• Title: FantasyHSI: Video-Generation-Centric 4D Human Synthesis In Any Scene through A Graph-based Multi-Agent Framework
• Abstract summary: Human-Scene Interaction (HSI) seeks to generate realistic human behaviors within complex environments.<n>HSI faces challenges in handling long-horizon, high-level tasks and generalizing to unseen scenes.<n>We introduce FantasyHSI, a novel HSI framework centered on video generation and multi-agent systems that operates without paired data.
• Score: 10.386464385736842
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human-Scene Interaction (HSI) seeks to generate realistic human behaviors within complex environments, yet it faces significant challenges in handling long-horizon, high-level tasks and generalizing to unseen scenes. To address these limitations, we introduce FantasyHSI, a novel HSI framework centered on video generation and multi-agent systems that operates without paired data. We model the complex interaction process as a dynamic directed graph, upon which we build a collaborative multi-agent system. This system comprises a scene navigator agent for environmental perception and high-level path planning, and a planning agent that decomposes long-horizon goals into atomic actions. Critically, we introduce a critic agent that establishes a closed-loop feedback mechanism by evaluating the deviation between generated actions and the planned path. This allows for the dynamic correction of trajectory drifts caused by the stochasticity of the generative model, thereby ensuring long-term logical consistency. To enhance the physical realism of the generated motions, we leverage Direct Preference Optimization (DPO) to train the action generator, significantly reducing artifacts such as limb distortion and foot-sliding. Extensive experiments on our custom SceneBench benchmark demonstrate that FantasyHSI significantly outperforms existing methods in terms of generalization, long-horizon task completion, and physical realism. Ours project page: https://fantasy-amap.github.io/fantasy-hsi/

Related papers

• G-MAPP: GPU-accelerated Multi-Agent Planning and Perception for Reactive Motion Generation [22.044839022400293]
  We propose a framework that does not compromise on runtime performance and world representations for perception and planning.<n>Our framework achieves up to a 5x speedup over the CPU version and successfully avoids collisions across both trivial and challenging physical world scenarios.
  arXiv  Detail & Related papers  (2026-06-10T18:28:24Z)
• InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement [15.184022425476256]
  Human-object-scene interactions (HOSI) generation has broad applications in embodied AI, simulation, and animation.<n>We propose a coarse-to-fine instruction-conditioned interaction generation framework that is aligned with the iterative denoising process of a consistency model.<n>Our method achieves state-of-the-art performance in both HOSI and HOI generation, and strong to unseen scenes.
  arXiv  Detail & Related papers  (2026-04-06T16:44:02Z)
• ONE-SHOT: Compositional Human-Environment Video Synthesis via Spatial-Decoupled Motion Injection and Hybrid Context Integration [49.72976665549397]
  We propose ONE-SHOT, a parameter-efficient framework for compositional human-environment video generation.<n>Our key insight is to factorize the generative process into disentangled signals. Specifically, we introduce a canonical-space injection mechanism that decouples human dynamics from environmental cues via cross-attention.<n>Our method significantly outperforms state-of-the-art methods, offering superior structural control and creative diversity for video synthesis.
  arXiv  Detail & Related papers  (2026-04-01T15:52:00Z)
• SceneTeract: Agentic Functional Affordances and VLM Grounding in 3D Scenes [49.117927512531715]
  We introduce SceneTeract, a framework that verifies 3D scene functionality under agent-specific constraints.<n> SceneTeract decomposes complex activities into sequences of atomic actions and validates each step against accessibility requirements.<n>We release the SceneTeract verification suite and data to bridge perception and physical reality in embodied 3D scene understanding.
  arXiv  Detail & Related papers  (2026-03-31T14:31:18Z)
• SAGE: Scalable Agentic 3D Scene Generation for Embodied AI [67.43935343696982]
  Existing scene-generation systems often rely on rule-based or task-specific pipelines, yielding artifacts and physically invalid scenes.<n>We present SAGE, an agentic framework that, given a user-specified embodied task, understands the intent and automatically generates simulation-ready environments at scale.<n>The resulting environments are realistic, diverse, and directly deployable in modern simulators for policy training.
  arXiv  Detail & Related papers  (2026-02-10T18:59:55Z)
• Generalizable Geometric Prior and Recurrent Spiking Feature Learning for Humanoid Robot Manipulation [90.90219129619344]
  This paper presents a novel R-prior-S, Recurrent Geometric-priormodal Policy with Spiking features.<n>To ground high-level reasoning in physical reality, we leverage lightweight 2D geometric inductive biases.<n>For the data efficiency issue in robotic action generation, we introduce a Recursive Adaptive Spiking Network.
  arXiv  Detail & Related papers  (2026-01-13T23:36:30Z)
• Astra: General Interactive World Model with Autoregressive Denoising [73.6594791733982]
  Astra is an interactive general world model that generates real-world futures for diverse scenarios.<n>We propose an autoregressive denoising architecture and use temporal causal attention to aggregate past observations.<n>Astra achieves interactive, consistent, and general long-term video prediction and supports various forms of interactions.
  arXiv  Detail & Related papers  (2025-12-09T18:59:57Z)
• Precise Action-to-Video Generation Through Visual Action Prompts [62.951609704196485]
  Action-driven video generation faces a precision-generality trade-off.<n>Agent-centric action signals provide precision at the cost of cross-domain transferability.<n>We "render" actions into precise visual prompts as domain-agnostic representations.
  arXiv  Detail & Related papers  (2025-08-18T17:12:28Z)
• SimGenHOI: Physically Realistic Whole-Body Humanoid-Object Interaction via Generative Modeling and Reinforcement Learning [6.255814224573073]
  SimGenHOI is a unified framework that combines the strengths of generative modeling and reinforcement learning to produce controllable and physically plausible HOI.<n>Our HOI generative model, based on Diffusion Transformers (DiT), predicts a set of key actions conditioned on text prompts, object geometry, sparse object waypoints, and the initial humanoid pose.<n>To ensure physical realism, we design a contact-aware whole-body control policy trained with reinforcement learning, which tracks the generated motions while correcting artifacts such as penetration and foot sliding.
  arXiv  Detail & Related papers  (2025-08-18T15:20:46Z)
• HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perception [57.37135310143126]
  HO SIG is a novel framework for synthesizing full-body interactions through hierarchical scene perception.<n>Our framework supports unlimited motion length through autoregressive generation and requires minimal manual intervention.<n>This work bridges the critical gap between scene-aware navigation and dexterous object manipulation.
  arXiv  Detail & Related papers  (2025-06-02T12:08:08Z)
• Hi-Dyna Graph: Hierarchical Dynamic Scene Graph for Robotic Autonomy in Human-Centric Environments [41.80879866951797]
  Hi-Dyna Graph is a hierarchical dynamic scene graph architecture that integrates persistent global layouts with localized dynamic semantics for embodied robotic autonomy.<n>An agent powered by large language models (LLMs) is employed to interpret the unified graph, infer latent task triggers, and generate executable instructions grounded in robotic affordances.
  arXiv  Detail & Related papers  (2025-05-30T03:35:29Z)
• Gen-C: Populating Virtual Worlds with Generative Crowds [2.1716667622896195]
  We introduce Generative Crowds (Gen-C), a generative framework that produces crowd scenarios capturing agent-agent and agent-environment interactions.<n>Gen-C employs a dual Variational Graph Autoencoder (VGAE) architecture that jointly learns connectivity patterns and node features conditioned on textual and structural signals.<n>We demonstrate the effectiveness of Gen-C on scenarios with diverse behaviors such as a University Campus and a Train Station.
  arXiv  Detail & Related papers  (2025-04-02T17:33:53Z)
• Scaling Up Dynamic Human-Scene Interaction Modeling [58.032368564071895]
  TRUMANS is the most comprehensive motion-captured HSI dataset currently available. It intricately captures whole-body human motions and part-level object dynamics. We devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length.
  arXiv  Detail & Related papers  (2024-03-13T15:45:04Z)
• SceneDM: Scene-level Multi-agent Trajectory Generation with Consistent Diffusion Models [10.057312592344507]
  We propose a novel framework based on diffusion models, called SceneDM, to generate joint and consistent future motions of all the agents in a scene. SceneDM achieves state-of-the-art results on the Sim Agents Benchmark.
  arXiv  Detail & Related papers  (2023-11-27T11:39:27Z)
• Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models [71.64318025625833]
  This paper presents a novel approach to generating the 3D motion of a human interacting with a target object. Our framework first generates a set of milestones and then synthesizes the motion along them. The experiments on the NSM, COUCH, and SAMP datasets show that our approach outperforms previous methods by a large margin in both quality and diversity.
  arXiv  Detail & Related papers  (2023-10-03T17:50:23Z)
• COPILOT: Human-Environment Collision Prediction and Localization from Egocentric Videos [62.34712951567793]
  The ability to forecast human-environment collisions from egocentric observations is vital to enable collision avoidance in applications such as VR, AR, and wearable assistive robotics. We introduce the challenging problem of predicting collisions in diverse environments from multi-view egocentric videos captured from body-mounted cameras. We propose a transformer-based model called COPILOT to perform collision prediction and localization simultaneously.
  arXiv  Detail & Related papers  (2022-10-04T17:49:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.