Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction

• URL: http://arxiv.org/abs/2604.27491v1
• Date: Thu, 30 Apr 2026 06:44:10 GMT
• Title: Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction
• Abstract summary: We propose Uni-HOI, a unified framework that learns the joint distribution among text, human motion, and object motion.<n>By leveraging large language models (LLMs) and two motion-specific vector quantized variational autoencoders (VQ-VAEs), we convert heterogeneous motion data into token sequences compatible with LLM inputs.<n>Experiments demonstrate that Uni-HOI achieves remarkable performances on multiple HOI-related tasks.
• Score: 6.292052507310665
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modeling 4D human-object interaction (HOI) is a compelling challenge in computer vision and an essential technology powering virtual and mixed-reality applications. While existing works have achieved promising results on specific HOI tasks-such as text-conditioned HOI generation and human motion generation from object motion, they typically rely on task-specific architectures and lack a unified framework capable of handling diverse conditional inputs. Building on this, we propose Uni-HOI, a unified framework that learns the joint distribution among text, human motion, and object motion. By leveraging large language models (LLMs) and two motion-specific vector quantized variational autoencoders (VQ-VAEs), we convert heterogeneous motion data into token sequences compatible with LLM inputs, enabling seamless integration and joint modeling of all three modalities. We introduce a two-stage training strategy: the first stage performs multi-task learning on a large-scale HOI dataset to capture the underlying correlations among the three modalities, while the second stage fine-tunes the model on specific tasks to further enhance performance. Extensive experiments demonstrate that Uni-HOI achieves remarkable performances on multiple HOI-related tasks including text-driven HOI generation, object motion-driven human motion generation (optionally with text) and human motion-driven object motion prediction within a unified framework.

Related papers

• MOCHI: Motion Enhancement of Collaborative Human-object Interactions [9.360852329235477]
  Collaborative human-object interaction shows and complex movements that require mutual anticipation and continuous adjustment between participants and the shared object.<n>We present MOCHI, a two-stage framework for enhancing noisy MHOI data.<n>Our approach first generates physically plausible hand grasps through optimization from noisy body input, producing grasps that are both physically plausible and semantically consistent with the body pose.<n>During the optimization process, we introduce optimization objectives to encode human-object and human-human interaction information within these single-person priors.
  arXiv  Detail & Related papers  (2026-06-16T17:58:44Z)
• Multimodal Priors-Augmented Text-Driven 3D Human-Object Interaction Generation [26.16137102387553]
  We address the challenging task of text-driven 3D human-object interaction (HOI) motion generation.<n>Existing methods primarily rely on a direct text-to-HOI mapping.<n>We propose MP-HOI, a novel framework grounded in four core insights.
  arXiv  Detail & Related papers  (2026-02-11T09:04:28Z)
• IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation [54.36300724708094]
  Assessment and refinement tasks act as crucial bridges to enable bidirectional knowledge flow between understanding and generation.<n>We introduce IRG-MotionLLM, the first model that seamlessly interleaves motion generation, assessment, and refinement to improve generation performance.
  arXiv  Detail & Related papers  (2025-12-11T15:16:06Z)
• Uni-Inter: Unifying 3D Human Motion Synthesis Across Diverse Interaction Contexts [59.78384600454231]
  We present Uni-Inter, a unified framework for human motion generation that supports a wide range of interaction scenarios.<n>Uni-Inter introduces the Unified Interactive Volume (UIV), a volumetric representation that encodes heterogeneous interactive entities into a shared spatial field.
  arXiv  Detail & Related papers  (2025-11-17T06:32:38Z)
• UniHM: Universal Human Motion Generation with Object Interactions in Indoor Scenes [26.71077287710599]
  We propose UniHM, a unified motion language model that leverages diffusion-based generation for scene-aware human motion.<n>UniHM is the first framework to support both Text-to-Motion and Text-to-Human-Object Interaction (HOI) in complex 3D scenes.<n>Our approach introduces three key contributions: (1) a mixed-motion representation that fuses continuous 6DoF motion with discrete local motion tokens to improve motion realism; (2) a novel Look-Up-Free Quantization VAE that surpasses traditional VQ-VAEs in both reconstruction accuracy and generative performance; and (3) an enriched version of
  arXiv  Detail & Related papers  (2025-05-19T07:02:12Z)
• EigenActor: Variant Body-Object Interaction Generation Evolved from Invariant Action Basis Reasoning [66.68366281305977]
  This paper explores a cross-modality synthesis task that infers 3D human-object interactions (HOIs) from a given text-based instruction.<n>Existing text-to-HOI synthesis methods mainly deploy a direct mapping from texts to object-specific 3D body motions.<n>We propose a novel body pose generation strategy for the text-to-HOI task: infer object-agnostic canonical body action first and then enrich object-specific interaction styles.
  arXiv  Detail & Related papers  (2025-03-01T07:15:10Z)
• COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Models [14.130327598928778]
  Large language models (LLMs) and hierarchical motion-specific vector-quantized variational autoencoders (VQ-VAEs) are proposed. Our framework generates realistic and diverse collaborative human-object-human interactions, outperforming state-of-the-art methods. Our work opens up new possibilities for modeling complex interactions in various domains, such as robotics, graphics and computer vision.
  arXiv  Detail & Related papers  (2024-09-30T17:02:13Z)
• Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection [116.21529970404653]
  We introduce SG2HOI+, a unified one-step model based on the Transformer architecture. Our approach employs two interactive hierarchical Transformers to seamlessly unify the tasks of SGG and HOI detection. Our approach achieves competitive performance when compared to state-of-the-art HOI methods.
  arXiv  Detail & Related papers  (2023-11-03T07:25:57Z)
• DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion [70.33381660741861]
  We present DiverseMotion, a new approach for synthesizing high-quality human motions conditioned on textual descriptions. We show that our DiverseMotion achieves the state-of-the-art motion quality and competitive motion diversity.
  arXiv  Detail & Related papers  (2023-09-04T05:43:48Z)
• TEMOS: Generating diverse human motions from textual descriptions [53.85978336198444]
  We address the problem of generating diverse 3D human motions from textual descriptions. We propose TEMOS, a text-conditioned generative model leveraging variational autoencoder (VAE) training with human motion data. We show that TEMOS framework can produce both skeleton-based animations as in prior work, as well more expressive SMPL body motions.
  arXiv  Detail & Related papers  (2022-04-25T14:53:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.