ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions

• URL: http://arxiv.org/abs/2603.25791v1
• Date: Thu, 26 Mar 2026 18:00:17 GMT
• Title: ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions
• Abstract summary: ArtHOI is an optimization-based framework that integrates and refines priors from multiple foundation models.<n>In particular, we introduce an Adaptive Sampling Refinement (ASR) method to optimize object's metric scale.<n>We also propose a Multimodal Large Language Model (MLLM) guided hand-object alignment method.
• Score: 48.84720445548848
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Existing hand-object interactions (HOI) methods are largely limited to rigid objects, while 4D reconstruction methods of articulated objects generally require pre-scanning the object or even multi-view videos. It remains an unexplored but significant challenge to reconstruct 4D human-articulated-object interactions from a single monocular RGB video. Fortunately, recent advancements in foundation models present a new opportunity to address this highly ill-posed problem. To this end, we introduce ArtHOI, an optimization-based framework that integrates and refines priors from multiple foundation models. Our key contribution is a suite of novel methodologies designed to resolve the inherent inaccuracies and physical unreality of these priors. In particular, we introduce an Adaptive Sampling Refinement (ASR) method to optimize object's metric scale and pose for grounding its normalized mesh in world space. Furthermore, we propose a Multimodal Large Language Model (MLLM) guided hand-object alignment method, utilizing contact reasoning information as constraints of hand-object mesh composition optimization. To facilitate a comprehensive evaluation, we also contribute two new datasets, ArtHOI-RGBD and ArtHOI-Wild. Extensive experiments validate the robustness and effectiveness of our ArtHOI across diverse objects and interactions. Project: https://arthoi-reconstruction.github.io.

Related papers

• High-Fidelity 4D Hand-Object Capture via Multi-View Spatiotemporal Tracking and Physics-Aware Gaussians [38.0162622592412]
  We propose a novel system for the robust and accurate reconstruction of hands and objects from synchronized and calibrated multi-view videos.<n>Our pipeline achieves highly robust, artifact-free reconstruction, providing an efficient foundation for automated 4D asset generation.
  arXiv  Detail & Related papers  (2026-06-14T16:36:49Z)
• MOSIV: Multi-Object System Identification from Videos [54.53609234233857]
  We propose a new framework that directly optimize for continuous, per-object material parameters using a differentiable simulator guided by geometric objectives derived from video.<n>We also present a new synthetic benchmark with contact-rich, multi-object interactions to facilitate evaluation.
  arXiv  Detail & Related papers  (2026-03-06T08:16:16Z)
• ArtLLM: Generating Articulated Assets via 3D LLM [19.814132638278547]
  ArtLLM is a novel framework for generating high-quality articulated assets directly from complete 3D meshes.<n>At its core is a 3D multimodal large language model trained on a large-scale articulation dataset.<n> Experiments show that ArtLLM significantly outperforms state-of-the-art methods in both part layout accuracy and joint prediction.
  arXiv  Detail & Related papers  (2026-03-01T15:07:46Z)
• ByteLoom: Weaving Geometry-Consistent Human-Object Interactions through Progressive Curriculum Learning [19.292101162897975]
  We introduce ByteLoom, a framework that generates realistic HOI videos with geometrically consistent object illustration.<n>We first propose an RCM-cache mechanism that leverages Relative Coordinate Maps (RCM) as a universal representation to maintain object's geometry consistency.<n>We then design a training curriculum that enhances model capabilities in a progressive style and relaxes the demand of hand mesh.
  arXiv  Detail & Related papers  (2025-12-28T09:38:36Z)
• Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model [72.90370736032115]
  We present a novel video Reenactment framework focusing on Human-Object Interaction (HOI) via an adaptive layout-instructed Diffusion model (Re-HOLD)<n>Our key insight is to employ specialized layout representation for hands and objects, respectively.<n>To further improve the generation quality of HOI, we design an interactive textural enhancement module for both hands and objects.
  arXiv  Detail & Related papers  (2025-03-21T08:40:35Z)
• ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting [66.29782808719301]
  Building articulated objects is a key challenge in computer vision.<n>Existing methods often fail to effectively integrate information across different object states.<n>We introduce ArtGS, a novel approach that leverages 3D Gaussians as a flexible and efficient representation.
  arXiv  Detail & Related papers  (2025-02-26T10:25:32Z)
• EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild [79.71523320368388]
  Our work aims to reconstruct hand-object interactions from a single-view image.<n>We first design a novel pipeline to estimate the underlying hand pose and object shape.<n>With the initial reconstruction, we employ a prior-guided optimization scheme.
  arXiv  Detail & Related papers  (2024-11-21T16:33:35Z)
• HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video [70.11702620562889]
  HOLD -- the first category-agnostic method that reconstructs an articulated hand and object jointly from a monocular interaction video. We develop a compositional articulated implicit model that can disentangled 3D hand and object from 2D images. Our method does not rely on 3D hand-object annotations while outperforming fully-supervised baselines in both in-the-lab and challenging in-the-wild settings.
  arXiv  Detail & Related papers  (2023-11-30T10:50:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.