Auto-Regressive Diffusion for Generating 3D Human-Object Interactions

• URL: http://arxiv.org/abs/2503.16801v1
• Date: Fri, 21 Mar 2025 02:25:59 GMT
• Title: Auto-Regressive Diffusion for Generating 3D Human-Object Interactions
• Abstract summary: Key challenge in HOI generation is maintaining interaction consistency in long sequences.<n>We propose an autoregressive diffusion model (ARDHOI) that predicts the next continuous token.<n>Our model has been evaluated on the OMOMO and BEHAVE datasets.
• Score: 5.587507490937267
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-driven Human-Object Interaction (Text-to-HOI) generation is an emerging field with applications in animation, video games, virtual reality, and robotics. A key challenge in HOI generation is maintaining interaction consistency in long sequences. Existing Text-to-Motion-based approaches, such as discrete motion tokenization, cannot be directly applied to HOI generation due to limited data in this domain and the complexity of the modality. To address the problem of interaction consistency in long sequences, we propose an autoregressive diffusion model (ARDHOI) that predicts the next continuous token. Specifically, we introduce a Contrastive Variational Autoencoder (cVAE) to learn a physically plausible space of continuous HOI tokens, thereby ensuring that generated human-object motions are realistic and natural. For generating sequences autoregressively, we develop a Mamba-based context encoder to capture and maintain consistent sequential actions. Additionally, we implement an MLP-based denoiser to generate the subsequent token conditioned on the encoded context. Our model has been evaluated on the OMOMO and BEHAVE datasets, where it outperforms existing state-of-the-art methods in terms of both performance and inference speed. This makes ARDHOI a robust and efficient solution for text-driven HOI tasks

Related papers

• MAD-HOI: Masked Autoregressive Diffusion for Generating Articulated Hand Object Interactions from Text [4.764824772285089]
  We present a model performing Masked Autoregression with Diffusion for HOI generation (MAD-HOI)<n>Our method starts by encoding hand and object motions in a continuous latent space while keeping them disentangled to maintain stream-wise control.<n> MAD-HOI is capable of motion generation for atomic and composite articulated sequences, conditioned motion completion and infilling, as well as EOM (End of Motion) prediction from a single training objective.
  arXiv  Detail & Related papers  (2026-08-10T19:30:59Z)
• SparseCtrl-HOI: Sparse Temporal Control for Human-Object Interaction Video Generation [29.683204938136758]
  Human-Object Interaction generation aims to synthesize realistic humans manipulating diverse objects.<n>Existing approaches to this problem rely on dense temporal guidance.<n>We introduce SparseCtrl-HOI, a novel sparse temporal control framework for HOI video generation.
  arXiv  Detail & Related papers  (2026-07-07T08:29:25Z)
• HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching [113.81911881001905]
  HO-Flow is a framework for synthesizing realistic hand-object motion sequences from texts and canoncial 3D objects.<n>It first employs an interaction-aware variational autoencoder to encode sequences of hand and object motions into a unified latent manifold.<n>It then leverages a masked flow matching model that combines auto-regressive temporal reasoning with continuous latent generation.
  arXiv  Detail & Related papers  (2026-04-12T22:06:11Z)
• InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement [15.184022425476256]
  Human-object-scene interactions (HOSI) generation has broad applications in embodied AI, simulation, and animation.<n>We propose a coarse-to-fine instruction-conditioned interaction generation framework that is aligned with the iterative denoising process of a consistency model.<n>Our method achieves state-of-the-art performance in both HOSI and HOI generation, and strong to unseen scenes.
  arXiv  Detail & Related papers  (2026-04-06T16:44:02Z)
• MIBURI: Towards Expressive Interactive Gesture Synthesis [62.45332399212876]
  Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions.<n>Existing solutions for ECAs produce rigid, low-diversity motions that are unsuitable for human-like interaction.<n>We present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue.
  arXiv  Detail & Related papers  (2026-03-03T18:59:51Z)
• FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation [23.19464039872024]
  FlowHOI is a flow-matching framework that generates temporally coherent HOI sequences.<n>We show that FlowHOI achieves the highest action recognition accuracy and a 1.7$times$ higher physics simulation success rate.
  arXiv  Detail & Related papers  (2026-02-13T20:46:08Z)
• HINT: Hierarchical Interaction Modeling for Autoregressive Multi-Human Motion Generation [55.73037290387896]
  We introduce HINT, the first autoregressive framework for multi-human motion generation with Hierarchical INTeraction modeling in diffusion.<n>First, HINT leverages a disentangled motion representation within a canonicalized latent space, decoupling local motion semantics from inter-person interactions.<n>Second, HINT adopts a sliding-window strategy for efficient online generation, and aggregates local within-window and global cross-window conditions to capture past human history, inter-person dependencies, and align with text guidance.
  arXiv  Detail & Related papers  (2026-01-28T08:47:23Z)
• ARIG: Autoregressive Interactive Head Generation for Real-time Conversations [15.886402427095515]
  Face-to-face communication, as a common human activity, motivates the research on interactive head generation.<n>Previous clip-wise generation paradigm or explicit listener/speaker generator-switching methods have limitations in future signal acquisition.<n>We propose an autoregressive (AR) based frame-wise framework called ARIG to realize the real-time generation with better interaction realism.
  arXiv  Detail & Related papers  (2025-07-01T06:38:14Z)
• Efficient Explicit Joint-level Interaction Modeling with Mamba for Text-guided HOI Generation [25.770855154106453]
  We introduce an Efficient Explicit Joint-level Interaction Model (EJIM) for generating text-guided human-object interactions. EJIM features a Dual-branch HOI Mamba that separately and efficiently models human object and motions. We show that EJIM surpasses previous works by a large margin while using only 5% of the inference time.
  arXiv  Detail & Related papers  (2025-03-29T15:23:21Z)
• Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression [23.99292102237088]
  We propose Heterogeneous Masked Autoregression (HMA) for modeling action-video dynamics.<n>After post-training, this model can be used as a video simulator for evaluating policies and generating synthetic data.
  arXiv  Detail & Related papers  (2025-02-06T18:38:26Z)
• InterDyn: Controllable Interactive Dynamics with Video Diffusion Models [50.38647583839384]
  We propose InterDyn, a framework that generates videos of interactive dynamics given an initial frame and a control signal encoding motion of a driving object or actor.<n>Our key insight is that large video generation models can act as both neurals and implicit physics simulators, having learned interactive dynamics from large-scale video data.
  arXiv  Detail & Related papers  (2024-12-16T13:57:02Z)
• HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objects [86.86284624825356]
  HIMO is a dataset of full-body human interacting with multiple objects. HIMO contains 3.3K 4D HOI sequences and 4.08M 3D HOI frames.
  arXiv  Detail & Related papers  (2024-07-17T07:47:34Z)
• Text-guided 3D Human Motion Generation with Keyframe-based Parallel Skip Transformer [62.29951737214263]
  Existing algorithms directly generate the full sequence which is expensive and prone to errors. We propose KeyMotion, that generates plausible human motion sequences corresponding to input text. We use a Variationalcoder (VAE) with Kullback-Leibler regularization to project the Autoencoder into a latent space. For the reverse diffusion, we propose a novel Parallel Skip Transformer that performs cross-modal attention between the design latents and text condition.
  arXiv  Detail & Related papers  (2024-05-24T11:12:37Z)
• HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models [42.62823339416957]
  We address the problem of generating realistic 3D human-object interactions (HOIs) driven by textual prompts. We first develop a dual-branch diffusion model (HOI-DM) to generate both human and object motions conditioned on the input text. We also develop an affordance prediction diffusion model (APDM) to predict the contacting area between the human and object.
  arXiv  Detail & Related papers  (2023-12-11T17:41:17Z)
• Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models [71.64318025625833]
  This paper presents a novel approach to generating the 3D motion of a human interacting with a target object. Our framework first generates a set of milestones and then synthesizes the motion along them. The experiments on the NSM, COUCH, and SAMP datasets show that our approach outperforms previous methods by a large margin in both quality and diversity.
  arXiv  Detail & Related papers  (2023-10-03T17:50:23Z)
• UDE: A Unified Driving Engine for Human Motion Generation [16.32286289924454]
  UDE is the first unified driving engine that enables generating human motion sequences from natural language or audio sequences. We evaluate our method on HumanML3DciteGuo_2022_CVPR and AIST++citeli 2021learn benchmarks.
  arXiv  Detail & Related papers  (2022-11-29T08:30:52Z)
• End-to-end Contextual Perception and Prediction with Interaction Transformer [79.14001602890417]
  We tackle the problem of detecting objects in 3D and forecasting their future motion in the context of self-driving. To capture their spatial-temporal dependencies, we propose a recurrent neural network with a novel Transformer architecture. Our model can be trained end-to-end, and runs in real-time.
  arXiv  Detail & Related papers  (2020-08-13T14:30:12Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.