OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation

• URL: http://arxiv.org/abs/2604.11804v2
• Date: Fri, 17 Apr 2026 08:08:09 GMT
• Title: OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation
• Abstract summary: Human-Object Interaction Video Generation (HOIVG) aims to synthesize high-quality human-object interaction videos conditioned on text, reference images, audio, and pose.<n>We present OmniShow, an end-to-end framework tailored for this practical yet challenging task.
• Score: 75.70865997182294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we study Human-Object Interaction Video Generation (HOIVG), which aims to synthesize high-quality human-object interaction videos conditioned on text, reference images, audio, and pose. This task holds significant practical value for automating content creation in real-world applications, such as e-commerce demonstrations, short video production, and interactive entertainment. However, existing approaches fail to accommodate all these requisite conditions. We present OmniShow, an end-to-end framework tailored for this practical yet challenging task, capable of harmonizing multimodal conditions and delivering industry-grade performance. To overcome the trade-off between controllability and quality, we introduce Unified Channel-wise Conditioning for efficient image and pose injection, and Gated Local-Context Attention to ensure precise audio-visual synchronization. To effectively address data scarcity, we develop a Decoupled-Then-Joint Training strategy that leverages a multi-stage training process with model merging to efficiently harness heterogeneous sub-task datasets. Furthermore, to fill the evaluation gap in this field, we establish HOIVG-Bench, a dedicated and comprehensive benchmark for HOIVG. Extensive experiments demonstrate that OmniShow achieves overall state-of-the-art performance across various multimodal conditioning settings, setting a solid standard for the emerging HOIVG task.

Related papers

• GEM: Generative Supervision Helps Embodied Intelligence [60.28190143133428]
  Embodied Vision-Language Models (VLMs) have demonstrated impressive performance and generalization in robotics.<n>A significant gap remains between the high-level semantic focus of standard text-guided pre-training paradigms and the low-level spatial and physical knowledge critical for execution in embodied environments.<n>We introduce GEM, a Generative-supervised Embodied vision-language Model designed to bridge this divide.
  arXiv  Detail & Related papers  (2026-05-27T14:39:42Z)
• Crab$^{+}$: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation [83.75249714794977]
  We present Crab$+$, a scalable and unified audio-visual scene understanding model.<n>On the data side, we introduce AV-UIE v2, a comprehensive Audio-Visual Unified Instruction-tuning dataset.<n>On the model side, we design a unified interface to align heterogeneous task formulations.<n>We successfully reverse the negative transfer trend, achieving positive transfer where multi-task learning surpasses single-task baselines in nearly 88% of tasks.
  arXiv  Detail & Related papers  (2026-03-04T14:43:57Z)
• Audio-Visual World Models: Towards Multisensory Imagination in Sight and Sound [5.591620304505415]
  This work presents the first formal framework for Audio-Visual World Models (AVWM)<n>It formulates multimodal environment simulation as a partially observable decision process with audio-visual observations, fine-grained actions, and task rewards.<n>We propose an Audio-Visual Conditional Transformer with a novel modality expert architecture that balances visual and auditory learning.
  arXiv  Detail & Related papers  (2025-11-30T13:11:56Z)
• HI-TransPA: Hearing Impairments Translation Personal Assistant [23.33416647487016]
  We introduce the Omni-Model paradigm into assistive technology and present HI-TransPA, an instruction-driven audio-visual personal assistant.<n>The model fuses indistinct speech with lip dynamics, enabling both translation and dialogue within a single multimodal framework.<n>Experiments on purpose-built HI-Dialogue dataset show that HI-TransPA achieves state-of-the-art performance in both literal accuracy and semantic fidelity.
  arXiv  Detail & Related papers  (2025-11-13T03:27:39Z)
• OmniMotion-X: Versatile Multimodal Whole-Body Motion Generation [52.579531290307926]
  This paper introduces OmniMotion-X, a versatile framework for whole-body human motion generation.<n> OmniMotion-X efficiently supports diverse multimodal tasks, including text-to-motion, music-to-dance, speech-to-gesture.<n>To enable high-quality multimodal training, we construct OmniMoCap-X, the largest unified multimodal motion dataset to date.
  arXiv  Detail & Related papers  (2025-10-22T17:25:33Z)
• EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfer [35.27100635173712]
  Vision-language-action (VLA) models increasingly rely on diverse training data to achieve robust generalization.<n>We propose Embodied Manipulation Media Adaptation (EMMA), a VLA policy enhancement framework that integrates a generative data engine with an effective training pipeline.<n>DreamTransfer enables text-controlled visual editing of robot videos, transforming foreground, background, and lighting conditions without compromising 3D structure or geometrical plausibility.<n>AdaMix is a hard-sample-aware training strategy that dynamically reweights training batches to focus optimization on perceptually or kinematically challenging samples.
  arXiv  Detail & Related papers  (2025-09-26T14:34:44Z)
• HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning [33.868900473146496]
  We present HuMo, a framework for collaborative multimodal control.<n>HuMo surpasses specialized state-of-the-art methods in sub-tasks.
  arXiv  Detail & Related papers  (2025-09-10T11:54:29Z)
• ASMR: Augmenting Life Scenario using Large Generative Models for Robotic Action Reflection [21.75681306780917]
  This paper introduces a novel framework focusing on data augmentation in robotic assistance scenarios.<n>It involves leveraging a sophisticated large language model to simulate potential conversations and environmental contexts.<n>The additionally generated data serves to refine the latest multimodal models, enabling them to more accurately determine appropriate actions.
  arXiv  Detail & Related papers  (2025-06-16T19:58:54Z)
• AsynFusion: Towards Asynchronous Latent Consistency Models for Decoupled Whole-Body Audio-Driven Avatars [71.90109867684025]
  Whole-body audio-driven avatar pose and expression generation is a critical task for creating lifelike digital humans.<n>We propose AsynFusion, a novel framework that leverages diffusion transformers to achieve cohesive expression and gesture synthesis.<n>AsynFusion achieves state-of-the-art performance in generating real-time, synchronized whole-body animations.
  arXiv  Detail & Related papers  (2025-05-21T03:28:53Z)
• One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion [38.16599550115468]
  We propose to leverage low-level vision tasks from digital photography fusion, allowing for effective feature interaction through pixel-level supervision.<n>The proposed GIFNet supports diverse fusion tasks, achieving high performance across both seen and unseen scenarios with a single model.
  arXiv  Detail & Related papers  (2025-02-27T07:55:19Z)
• EVA: An Embodied World Model for Future Video Anticipation [30.721105710709008]
  Video generation models have made significant progress in simulating future states, showcasing their potential as world simulators in embodied scenarios.<n>Existing models often lack robust understanding, limiting their ability to perform multi-step predictions or handle Out-of-Distribution (OOD) scenarios.<n>We propose the Reflection of Generation (RoG), a set of intermediate reasoning strategies designed to enhance video prediction.
  arXiv  Detail & Related papers  (2024-10-20T18:24:00Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning [54.67880602409801]
  In this paper, we study the problem of pre-training world models with abundant in-the-wild videos for efficient learning of visual control tasks. We introduce Contextualized World Models (ContextWM) that explicitly separate context and dynamics modeling. Our experiments show that in-the-wild video pre-training equipped with ContextWM can significantly improve the sample efficiency of model-based reinforcement learning.
  arXiv  Detail & Related papers  (2023-05-29T14:29:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.