Real-Time Human-Centric World Modeling for Upper-Body Human-Object Interaction

• URL: http://arxiv.org/abs/2607.23517v1
• Date: Sun, 26 Jul 2026 07:34:14 GMT
• Title: Real-Time Human-Centric World Modeling for Upper-Body Human-Object Interaction
• Abstract summary: We present a real-time human-centric world model for upper-body interactive generation, aiming to synthesize coherent local world dynamics centered on a person.<n>We adopt a continuous-discrete joint control scheme with two complementary components: a continuous human state and a discrete interaction state.<n>Experiments demonstrate improved fine-grained motion fidelity, more hand-object coordination, and effective real-time interaction.
• Score: 15.252158606421752
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a real-time human-centric world model for upper-body interactive generation, aiming to synthesize coherent local world dynamics centered on a person, where coordinated body, hand, and facial motions evolve jointly with controllable human-object discrete interaction. To this end, we adopt a continuous-discrete joint control scheme with two complementary components: a continuous human state and a discrete interaction state. For continuous human-state control, we introduce a unified implicit representation based on multi-scale motion encoding, in which motion latents from the upper body, hands, and face are fused into a shared latent space. This multi-scale design improves expressiveness across different spatial scales, captures fine-grained human dynamics more effectively, and enables direct control without explicit retargeting. For discrete object interaction-state control, we represent object contact using a small set of language-encoded discrete interaction states, where text serves as an explicit interaction-state command, such as \emph{no contact} or \emph{grasp}, rather than an open-ended generation prompt, and we further construct a dedicated rendering pipeline for human-object interaction data to supervise such discrete interaction states. By combining continuous implicit human-state control with discrete interaction-state control, our model enables precise modeling of how a person moves and interacts with the local environment, including controllable changes to nearby scene states. Finally, we distill the model for efficient streaming real-time inference, achieving 25 FPS on two H100 GPUs. Experiments demonstrate improved fine-grained motion fidelity, more realistic hand-object coordination, and effective real-time interaction, establishing a practical step beyond motion reproduction toward real-time human-centric world modeling.

Related papers

• GNOCHI: Generative Neural mOdel for Close Human-Human Interactions [7.759879304243429]
  This paper introduces a novel generative model for close 3D human-human interactions using a conditional variational autoencoder (cVAE)<n>We address two underlying long-standing challenges in the field of human-human interaction: data scarcity and collision awareness.<n>We demonstrate a wide variety of plausible and physically correct interactions, not possible to generate with current state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-11T17:19:32Z)
• AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization [79.26935895370191]
  AnchorWorld is a framework that advances egocentric simulation through enhanced interaction integrity.<n> auxiliary training supervision incorporates viewpoints decoupled from the agent's first-person sensorium.<n>We propose a simple yet effective mechanism for customizing self-evolving worlds.
  arXiv  Detail & Related papers  (2026-06-05T14:43:13Z)
• SocialMirror: Reconstructing 3D Human Interaction Behaviors from Monocular Videos with Semantic and Geometric Guidance [49.69016078147708]
  Accurately reconstructing human behavior in close-interaction scenarios is crucial for enabling realistic virtual interactions in augmented reality.<n>We propose SocialMirror, a diffusion-based framework that integrates semantic and geometric cues to effectively address these issues.<n>SocialMirror achieves state-of-the-art performance in reconstructing interactive human meshes, demonstrating strong generalization across unseen datasets and in-the-wild scenarios.
  arXiv  Detail & Related papers  (2026-04-15T07:41:52Z)
• ReMoGen: Real-time Human Interaction-to-Reaction Generation via Modular Learning from Diverse Data [38.85809416362408]
  ReMoGen is a modular learning framework for real-time interaction-to-reaction generation.<n>It generates the ego's future motion from dynamic multi-source cues, including others' actions, scene geometry, and optional high-level semantic inputs.<n>ReMoGen produces high-quality, coherent, and responsive reactions, while generalizing effectively across diverse interaction scenarios.
  arXiv  Detail & Related papers  (2026-04-01T16:12:23Z)
• Interact2Ar: Full-Body Human-Human Interaction Generation via Autoregressive Diffusion Models [80.28579390566298]
  We introduce Interact2Ar, a text-conditioned autoregressive diffusion model for generating full-body, human-human interactions.<n>Hand kinematics are incorporated through dedicated parallel branches, enabling high-fidelity full-body generation.<n>Our model enables a series of downstream applications, including temporal motion composition, real-time adaptation to disturbances, and extension beyond dyadic to multi-person scenarios.
  arXiv  Detail & Related papers  (2025-12-22T18:59:50Z)
• Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis [51.95817740348585]
  Human-X is a novel framework designed to enable immersive and physically plausible human interactions across diverse entities.<n>Our method jointly predicts actions and reactions in real-time using an auto-regressive reaction diffusion planner.<n>Our framework is validated in real-world applications, including virtual reality interface for human-robot interaction.
  arXiv  Detail & Related papers  (2025-08-04T06:35:48Z)
• Controllable Human-Object Interaction Synthesis [77.56877961681462]
  We propose Controllable Human-Object Interaction Synthesis (CHOIS) to generate synchronized object motion and human motion in 3D scenes. Here, language descriptions inform style and intent, and waypoints, which can be effectively extracted from high-level planning, ground the motion in the scene. Our module seamlessly integrates with a path planning module, enabling the generation of long-term interactions in 3D environments.
  arXiv  Detail & Related papers  (2023-12-06T21:14:20Z)
• InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint [67.6297384588837]
  We introduce a novel controllable motion generation method, InterControl, to encourage the synthesized motions maintaining the desired distance between joint pairs. We demonstrate that the distance between joint pairs for human-wise interactions can be generated using an off-the-shelf Large Language Model.
  arXiv  Detail & Related papers  (2023-11-27T14:32:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.