InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement

• URL: http://arxiv.org/abs/2604.04843v1
• Date: Mon, 06 Apr 2026 16:44:02 GMT
• Title: InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement
• Abstract summary: Human-object-scene interactions (HOSI) generation has broad applications in embodied AI, simulation, and animation.<n>We propose a coarse-to-fine instruction-conditioned interaction generation framework that is aligned with the iterative denoising process of a consistency model.<n>Our method achieves state-of-the-art performance in both HOSI and HOI generation, and strong to unseen scenes.
• Score: 15.184022425476256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human-object-scene interactions (HOSI) generation has broad applications in embodied AI, simulation, and animation. Unlike human-object interaction (HOI) and human-scene interaction (HSI), HOSI generation requires reasoning over dynamic object-scene changes, yet suffers from limited annotated data. To address these issues, we propose a coarse-to-fine instruction-conditioned interaction generation framework that is explicitly aligned with the iterative denoising process of a consistency model. In particular, we adopt a dynamic perception strategy that leverages trajectories from the preceding refinement to update scene context and condition subsequent refinement at each denoising step of consistency model, yielding consistent interactions. To further reduce physical artifacts, we introduce a bump-aware guidance that mitigates collisions and penetrations during sampling without requiring fine-grained scene geometry, enabling real-time generation. To overcome data scarcity, we design a hybrid training startegy that synthesizes pseudo-HOSI samples by injecting voxelized scene occupancy into HOI datasets and jointly trains with high-fidelity HSI data, allowing interaction learning while preserving realistic scene awareness. Extensive experiments demonstrate that our method achieves state-of-the-art performance in both HOSI and HOI generation, and strong generalization to unseen scenes. Project page: https://yudezou.github.io/InfBaGel-page/

Related papers

• PhyScene3D: Physically Consistent Interactive 3D Tabletop Scene Generation [75.9028457575434]
  Generating 3D tabletop scenes is a fundamental yet underexplored problem for interactive and generalist robotic learning.<n>Here, we introduce PhyScene3D, a framework that reformulates generation as a Human-Mimetic Constructive Process.<n>Experiments demonstrate that PhyScene3D outperforms state-of-the-art approaches in both semantic accuracy and physical validity.
  arXiv  Detail & Related papers  (2026-06-01T03:56:32Z)
• HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching [113.81911881001905]
  HO-Flow is a framework for synthesizing realistic hand-object motion sequences from texts and canoncial 3D objects.<n>It first employs an interaction-aware variational autoencoder to encode sequences of hand and object motions into a unified latent manifold.<n>It then leverages a masked flow matching model that combines auto-regressive temporal reasoning with continuous latent generation.
  arXiv  Detail & Related papers  (2026-04-12T22:06:11Z)
• GenHOI: Towards Object-Consistent Hand-Object Interaction with Temporally Balanced and Spatially Selective Object Injection [54.879037588415656]
  GenHOI is a lightweight augmentation to pretrained video generation models.<n>It injects reference-object information in a temporally balanced and spatially selective manner.<n>GenHOI significantly outperforms state-of-the-art HOI reenactment and all-in-one video editing methods.
  arXiv  Detail & Related papers  (2026-03-06T09:01:09Z)
• Learning Whole-Body Human-Humanoid Interaction from Human-Human Demonstrations [63.80827184637476]
  We introduce D-STAR, a hierarchical policy that disentangles when to act from where to act.<n>We validate our framework through extensive and rigorous simulations.
  arXiv  Detail & Related papers  (2026-01-14T14:37:06Z)
• FantasyHSI: Video-Generation-Centric 4D Human Synthesis In Any Scene through A Graph-based Multi-Agent Framework [10.386464385736842]
  Human-Scene Interaction (HSI) seeks to generate realistic human behaviors within complex environments.<n>HSI faces challenges in handling long-horizon, high-level tasks and generalizing to unseen scenes.<n>We introduce FantasyHSI, a novel HSI framework centered on video generation and multi-agent systems that operates without paired data.
  arXiv  Detail & Related papers  (2025-09-01T08:20:50Z)
• SimGenHOI: Physically Realistic Whole-Body Humanoid-Object Interaction via Generative Modeling and Reinforcement Learning [6.255814224573073]
  SimGenHOI is a unified framework that combines the strengths of generative modeling and reinforcement learning to produce controllable and physically plausible HOI.<n>Our HOI generative model, based on Diffusion Transformers (DiT), predicts a set of key actions conditioned on text prompts, object geometry, sparse object waypoints, and the initial humanoid pose.<n>To ensure physical realism, we design a contact-aware whole-body control policy trained with reinforcement learning, which tracks the generated motions while correcting artifacts such as penetration and foot sliding.
  arXiv  Detail & Related papers  (2025-08-18T15:20:46Z)
• Auto-Regressive Diffusion for Generating 3D Human-Object Interactions [5.587507490937267]
  Key challenge in HOI generation is maintaining interaction consistency in long sequences.<n>We propose an autoregressive diffusion model (ARDHOI) that predicts the next continuous token.<n>Our model has been evaluated on the OMOMO and BEHAVE datasets.
  arXiv  Detail & Related papers  (2025-03-21T02:25:59Z)
• Scaling Up Dynamic Human-Scene Interaction Modeling [58.032368564071895]
  TRUMANS is the most comprehensive motion-captured HSI dataset currently available. It intricately captures whole-body human motions and part-level object dynamics. We devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length.
  arXiv  Detail & Related papers  (2024-03-13T15:45:04Z)
• Local-Global Information Interaction Debiasing for Dynamic Scene Graph Generation [51.92419880088668]
  We propose a novel DynSGG model based on multi-task learning, DynSGG-MTL, which introduces the local interaction information and global human-action interaction information. Long-temporal human actions supervise the model to generate multiple scene graphs that conform to the global constraints and avoid the model being unable to learn the tail predicates.
  arXiv  Detail & Related papers  (2023-08-10T01:24:25Z)
• ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation [135.10594078615952]
  We introduce ACID, an action-conditional visual dynamics model for volumetric deformable objects. A benchmark contains over 17,000 action trajectories with six types of plush toys and 78 variants. Our model achieves the best performance in geometry, correspondence, and dynamics predictions.
  arXiv  Detail & Related papers  (2022-03-14T04:56:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.