Zero-shot Reconstruction of In-Scene Object Manipulation from Video

• URL: http://arxiv.org/abs/2512.19684v1
• Date: Mon, 22 Dec 2025 18:58:29 GMT
• Title: Zero-shot Reconstruction of In-Scene Object Manipulation from Video
• Abstract summary: We build the first system to address the problem of reconstructing in-scene object manipulation from a monocular RGB video.<n>It is challenging due to ill-posed scene reconstruction, ambiguous hand-object depth, and the need for physically plausible interactions.
• Score: 47.13702503259619
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We build the first system to address the problem of reconstructing in-scene object manipulation from a monocular RGB video. It is challenging due to ill-posed scene reconstruction, ambiguous hand-object depth, and the need for physically plausible interactions. Existing methods operate in hand centric coordinates and ignore the scene, hindering metric accuracy and practical use. In our method, we first use data-driven foundation models to initialize the core components, including the object mesh and poses, the scene point cloud, and the hand poses. We then apply a two-stage optimization that recovers a complete hand-object motion from grasping to interaction, which remains consistent with the scene information observed in the input video.

Related papers

• ComPose: When to Trust Hands for Object Pose Tracking [44.085148707189035]
  ComPose is a 6DoF object tracking framework designed for hand-aware object pose estimation from RGB video.<n>Our method harmonizes hand motions as a textitcomplementary cue for object tracking.<n>Result can effectively transfer to downstream robot manipulation by enabling robots to reconstruct human actions from online videos.
  arXiv  Detail & Related papers  (2026-05-22T11:39:49Z)
• sim2art: Accurate Articulated Object Modeling from a Single Video using Synthetic Training Data Only [20.99905717289565]
  We present the first data-driven approach that jointly predicts part segmentation and joint parameters from monocular video captured with a freely moving camera.<n>Our method demonstrates strong generalization to real-world objects, offering a scalable and practical solution for articulated object understanding.<n>Our approach operates directly on casually recorded video, making it suitable for real-time applications in dynamic environments.
  arXiv  Detail & Related papers  (2025-12-08T16:38:30Z)
• VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Video [60.63575135514847]
  Building digital twins of articulated objects from monocular video presents an essential challenge in computer vision.<n>We introduce VideoArtGS, a novel approach that reconstructs high-fidelity digital twins of articulated objects from monocular video.<n>VideoArtGS demonstrates state-of-the-art performance in articulation and mesh reconstruction, reducing the reconstruction error by about two orders of magnitude compared to existing methods.
  arXiv  Detail & Related papers  (2025-09-22T11:52:02Z)
• Object Pose Estimation through Dexterous Touch [27.99244228962149]
  Our approach uses sensorimotor exploration to actively control a robot hand to interact with the object.<n>We show that our method can actively explore an object's surface to identify critical pose features without prior knowledge of the object's geometry.
  arXiv  Detail & Related papers  (2025-09-16T23:25:05Z)
• iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videos [52.398752421673144]
  We focus on motion analysis and part-level segmentation of an articulated object from a casually captured RGBD video shot with a hand-held camera.<n>A casually captured video of an interaction with an articulated object is easy to obtain at scale using smartphones.<n>We introduce iTACO: a coarse-to-fine framework that infers joint parameters and segments movable parts of the object from a dynamic RGBD video.
  arXiv  Detail & Related papers  (2025-06-10T01:41:46Z)
• HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video [70.11702620562889]
  HOLD -- the first category-agnostic method that reconstructs an articulated hand and object jointly from a monocular interaction video. We develop a compositional articulated implicit model that can disentangled 3D hand and object from 2D images. Our method does not rely on 3D hand-object annotations while outperforming fully-supervised baselines in both in-the-lab and challenging in-the-wild settings.
  arXiv  Detail & Related papers  (2023-11-30T10:50:35Z)
• SHOWMe: Benchmarking Object-agnostic Hand-Object 3D Reconstruction [13.417086460511696]
  We introduce the SHOWMe dataset which consists of 96 videos, annotated with real and detailed hand-object 3D textured meshes. We consider a rigid hand-object scenario, in which the pose of the hand with respect to the object remains constant during the whole video sequence. This assumption allows us to register sub-millimetre-precise groundtruth 3D scans to the image sequences in SHOWMe.
  arXiv  Detail & Related papers  (2023-09-19T16:48:29Z)
• Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis [76.72505510632904]
  We present Total-Recon, the first method to reconstruct deformable scenes from long monocular RGBD videos. Our method hierarchically decomposes the scene into the background and objects, whose motion is decomposed into root-body motion and local articulations.
  arXiv  Detail & Related papers  (2023-04-24T17:59:52Z)
• TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinement [42.3418874174372]
  We present TOCH, a method for refining incorrect 3D hand-object interaction sequences using a data prior. We learn a latent manifold of plausible TOCH fields with a temporal denoising auto-encoder. Experiments demonstrate that TOCH outperforms state-of-the-art 3D hand-object interaction models.
  arXiv  Detail & Related papers  (2022-05-16T20:41:45Z)
• NeuralDiff: Segmenting 3D objects that move in egocentric videos [92.95176458079047]
  We study the problem of decomposing the observed 3D scene into a static background and a dynamic foreground. This task is reminiscent of the classic background subtraction problem, but is significantly harder because all parts of the scene, static and dynamic, generate a large apparent motion. In particular, we consider egocentric videos and further separate the dynamic component into objects and the actor that observes and moves them.
  arXiv  Detail & Related papers  (2021-10-19T12:51:35Z)
• Towards unconstrained joint hand-object reconstruction from RGB videos [81.97694449736414]
  Reconstructing hand-object manipulations holds a great potential for robotics and learning from human demonstrations. We first propose a learning-free fitting approach for hand-object reconstruction which can seamlessly handle two-hand object interactions.
  arXiv  Detail & Related papers  (2021-08-16T12:26:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.