Reconstructing Humans and Objects in Interaction using Large Reconstruction Models

• URL: http://arxiv.org/abs/2608.27407v1
• Date: Thu, 27 Aug 2026 17:35:46 GMT
• Title: Reconstructing Humans and Objects in Interaction using Large Reconstruction Models
• Abstract summary: Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI.<n>We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image.
• Score: 20.36962850081718
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue. We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies the reconstruction procedure, reframing the problem as interpreting the LRM mesh: we segment it into human and object components, fit a parametric body model to the human part, and optionally align an object template to the object part (if such a template is available). MILO achieves strong reconstruction accuracy and outperforms existing baselines across multiple benchmarks and interaction scenarios. Our code is available at https://ac5113.github.io/MILO.

Related papers

• Human Interaction-Aware 3D Reconstruction from a Single Image [25.835242245333294]
  Existing methods mostly focus on single individuals and thus fail in multi-human scenes.<n>We introduce a holistic method that explicitly models both group- and instance-level information.<n>We produce physically plausible, high-fidelity 3D reconstructions of interacting people from a single image.
  arXiv  Detail & Related papers  (2026-04-07T05:09:41Z)
• ArtLLM: Generating Articulated Assets via 3D LLM [19.814132638278547]
  ArtLLM is a novel framework for generating high-quality articulated assets directly from complete 3D meshes.<n>At its core is a 3D multimodal large language model trained on a large-scale articulation dataset.<n> Experiments show that ArtLLM significantly outperforms state-of-the-art methods in both part layout accuracy and joint prediction.
  arXiv  Detail & Related papers  (2026-03-01T15:07:46Z)
• InteractVLM: 3D Interaction Reasoning from 2D Foundational Models [85.76211596755151]
  We introduce InteractVLM, a novel method to estimate 3D contact points on human bodies and objects from single in-the-wild images.<n>Existing methods rely on 3D contact annotations collected via expensive motion-capture systems or tedious manual labeling.<n>We propose a new task called Semantic Human Contact estimation, where human contact predictions are conditioned explicitly on object semantics.
  arXiv  Detail & Related papers  (2025-04-07T17:59:33Z)
• Zero-Shot Human-Object Interaction Synthesis with Multimodal Priors [31.277540988829976]
  This paper proposes a novel zero-shot HOI synthesis framework without relying on end-to-end training on currently limited 3D HOI datasets.<n>We employ pre-trained human pose estimation models to extract human poses and introduce a generalizable category-level 6-DoF estimation method to obtain the object poses from 2D HOI images.
  arXiv  Detail & Related papers  (2025-03-25T23:55:47Z)
• EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild [79.71523320368388]
  Our work aims to reconstruct hand-object interactions from a single-view image.<n>We first design a novel pipeline to estimate the underlying hand pose and object shape.<n>With the initial reconstruction, we employ a prior-guided optimization scheme.
  arXiv  Detail & Related papers  (2024-11-21T16:33:35Z)
• StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset [56.71580976007712]
  We propose to use the Human-Object Offset between anchors which are densely sampled from the surface of human mesh and object mesh to represent human-object spatial relation. Based on this representation, we propose Stacked Normalizing Flow (StackFLOW) to infer the posterior distribution of human-object spatial relations from the image. During the optimization stage, we finetune the human body pose and object 6D pose by maximizing the likelihood of samples.
  arXiv  Detail & Related papers  (2024-07-30T04:57:21Z)
• HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video [70.11702620562889]
  HOLD -- the first category-agnostic method that reconstructs an articulated hand and object jointly from a monocular interaction video. We develop a compositional articulated implicit model that can disentangled 3D hand and object from 2D images. Our method does not rely on 3D hand-object annotations while outperforming fully-supervised baselines in both in-the-lab and challenging in-the-wild settings.
  arXiv  Detail & Related papers  (2023-11-30T10:50:35Z)
• Full-Body Articulated Human-Object Interaction [61.01135739641217]
  CHAIRS is a large-scale motion-captured f-AHOI dataset consisting of 16.2 hours of versatile interactions. CHAIRS provides 3D meshes of both humans and articulated objects during the entire interactive process. By learning the geometrical relationships in HOI, we devise the very first model that leverage human pose estimation.
  arXiv  Detail & Related papers  (2022-12-20T19:50:54Z)
• CHORE: Contact, Human and Object REconstruction from a single RGB image [40.817960406002506]
  CHORE is a novel method that learns to jointly reconstruct the human and the object from a single RGB image. We compute a neural reconstruction of human and object represented implicitly with two unsigned distance fields. Experiments show that our joint reconstruction learned with the proposed strategy significantly outperforms the SOTA.
  arXiv  Detail & Related papers  (2022-04-05T18:38:06Z)
• DemoGrasp: Few-Shot Learning for Robotic Grasping with Human Demonstration [42.19014385637538]
  We propose to teach a robot how to grasp an object with a simple and short human demonstration. We first present a small sequence of RGB-D images displaying a human-object interaction. This sequence is then leveraged to build associated hand and object meshes that represent the interaction.
  arXiv  Detail & Related papers  (2021-12-06T08:17:12Z)
• Amodal 3D Reconstruction for Robotic Manipulation via Stability and Connectivity [3.359622001455893]
  Learning-based 3D object reconstruction enables single- or few-shot estimation of 3D object models. Existing 3D reconstruction techniques optimize for visual reconstruction fidelity, typically measured by chamfer distance or voxel IOU. We propose ARM, an amodal 3D reconstruction system that introduces a stability prior over object shapes, (2) a connectivity prior, and (3) a multi-channel input representation.
  arXiv  Detail & Related papers  (2020-09-28T08:52:54Z)
• HMOR: Hierarchical Multi-Person Ordinal Relations for Monocular Multi-Person 3D Pose Estimation [54.23770284299979]
  This paper introduces a novel form of supervision - Hierarchical Multi-person Ordinal Relations (HMOR) HMOR encodes interaction information as the ordinal relations of depths and angles hierarchically. An integrated top-down model is designed to leverage these ordinal relations in the learning process. The proposed method significantly outperforms state-of-the-art methods on publicly available multi-person 3D pose datasets.
  arXiv  Detail & Related papers  (2020-08-01T07:53:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.