MVHOI: Bridge Multi-view Condition to Complex Human-Object Interaction Video Reenactment via 3D Foundation Model

• URL: http://arxiv.org/abs/2603.14686v1
• Date: Mon, 16 Mar 2026 00:43:38 GMT
• Title: MVHOI: Bridge Multi-view Condition to Complex Human-Object Interaction Video Reenactment via 3D Foundation Model
• Abstract summary: Human-Object Interaction (HOI) video reenactment with realistic motion remains a frontier in digital human creation.<n>We propose MVHOI, a two-stage HOI video reenactment framework that bridges multi-view reference conditions and video foundation models.<n>Our framework shows superior performance in generating long-duration HOI videos with intricate object manipulations.
• Score: 83.12400850233034
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Human-Object Interaction (HOI) video reenactment with realistic motion remains a frontier in expressive digital human creation. Existing approaches primarily handle simple image-plane motion (e.g., in-plane translations), struggling with complex non-planar manipulations like out-of-plane reorientation. In this paper, we propose MVHOI, a two-stage HOI video reenactment framework that bridges multi-view reference conditions and video foundation models via a 3D Foundation Model (3DFM). The 3DFM first produces view-consistent object priors conditioned on implicit motion dynamics across novel viewpoints. A controllable video generation model then synthesizes high-fidelity object texture by incorporating multi-view reference images, ensuring appearance consistency via a reasonable retrieval mechanism. By enabling these two stages to mutually reinforce one another during the inference phase, our framework shows superior performance in generating long-duration HOI videos with intricate object manipulations. Extensive experiments show substantial improvements over prior approaches, especially for HOI with complex 3D object manipulations.

Related papers

• Towards Realistic and Consistent Orbital Video Generation via 3D Foundation Priors [61.34273238077091]
  We present a novel method for generating orbital videos from a single image of an object.<n>Our method achieves superior visual quality, shape realism and multi-view consistency compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-04-14T05:35:46Z)
• ViHOI: Human-Object Interaction Synthesis with Visual Priors [21.133510317025635]
  ViHOI is a novel framework that enables diffusion-based generative models to leverage rich, task-specific priors from 2D images.<n>Our framework is trained on motion-rendered images from the dataset to ensure strict semantic alignment between visual inputs and motion sequences.
  arXiv  Detail & Related papers  (2026-03-25T15:02:06Z)
• SyncMV4D: Synchronized Multi-view Joint Diffusion of Appearance and Motion for Hand-Object Interaction Synthesis [47.61773799705708]
  We introduce SyncMV4D, the first model that jointly generates synchronized multi-view HOI videos and 4D motions.<n>Our method demonstrates superior performance to state-of-the-art alternatives in visual realism, motion plausibility, and multi-view consistency.
  arXiv  Detail & Related papers  (2025-11-24T17:14:19Z)
• View-Consistent Diffusion Representations for 3D-Consistent Video Generation [60.68052293389281]
  Current generated videos still contain visual artifacts arising from 3D inconsistencies.<n>We propose ViCoDR, a new approach for improving the 3D consistency of video models by learning multi-view consistent diffusion representations.
  arXiv  Detail & Related papers  (2025-11-24T11:16:55Z)
• VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Video [60.63575135514847]
  Building digital twins of articulated objects from monocular video presents an essential challenge in computer vision.<n>We introduce VideoArtGS, a novel approach that reconstructs high-fidelity digital twins of articulated objects from monocular video.<n>VideoArtGS demonstrates state-of-the-art performance in articulation and mesh reconstruction, reducing the reconstruction error by about two orders of magnitude compared to existing methods.
  arXiv  Detail & Related papers  (2025-09-22T11:52:02Z)
• Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control [72.00655365269]
  We present RoboMaster, a novel framework that models inter-object dynamics through a collaborative trajectory formulation.<n>Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction.<n>Our method outperforms existing approaches, establishing new state-of-the-art performance in trajectory-controlled video generation for robotic manipulation.
  arXiv  Detail & Related papers  (2025-06-02T17:57:06Z)
• ReVision: High-Quality, Low-Cost Video Generation with Explicit 3D Physics Modeling for Complex Motion and Interaction [22.420752010237052]
  We introduce ReVision, a plug-and-play framework that explicitly integrates parameterized 3D physical knowledge into a conditional video generation model.<n>We validate the effectiveness of our approach on Stable Video Diffusion, where ReVision significantly improves motion fidelity and coherence.<n>Our results suggest that, by incorporating 3D physical knowledge, even a relatively small video diffusion model can generate complex motions and interactions with greater realism and controllability.
  arXiv  Detail & Related papers  (2025-04-30T17:59:56Z)
• CDI3D: Cross-guided Dense-view Interpolation for 3D Reconstruction [25.468907201804093]
  Large Reconstruction Models (LRMs) have shown great promise in leveraging multi-view images generated by 2D diffusion models to extract 3D content.<n>However, 2D diffusion models often struggle to produce dense images with strong multi-view consistency.<n>We present CDI3D, a feed-forward framework designed for efficient, high-quality image-to-3D generation with view.
  arXiv  Detail & Related papers  (2025-03-11T03:08:43Z)
• Shape of Motion: 4D Reconstruction from a Single Video [42.42669078777769]
  We introduce a method for reconstructing generic dynamic scenes, featuring explicit, persistent 3D motion trajectories in the world coordinate frame.<n>First, we exploit the low-dimensional structure of 3D motion by representing scene motion with a compact set of SE(3) motion bases.<n>Second, we take advantage of off-the-shelf data-driven priors such as monocular depth maps and long-range 2D tracks, and devise a method to effectively consolidate these noisy supervisory signals.
  arXiv  Detail & Related papers  (2024-07-18T17:59:08Z)
• Decoupling Dynamic Monocular Videos for Dynamic View Synthesis [50.93409250217699]
  We tackle the challenge of dynamic view synthesis from dynamic monocular videos in an unsupervised fashion. Specifically, we decouple the motion of the dynamic objects into object motion and camera motion, respectively regularized by proposed unsupervised surface consistency and patch-based multi-view constraints.
  arXiv  Detail & Related papers  (2023-04-04T11:25:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.