MessyKitchens: Contact-rich object-level 3D scene reconstruction

• URL: http://arxiv.org/abs/2603.16868v1
• Date: Tue, 17 Mar 2026 17:59:51 GMT
• Title: MessyKitchens: Contact-rich object-level 3D scene reconstruction
• Abstract summary: We introduce MessyKitchens, a new dataset with real-world scenes featuring cluttered environments.<n>We build on the recent SAM 3D approach for single-object reconstruction and extend it with Multi-Object Decoder (MOD) for joint object-level scene reconstruction.
• Score: 26.192713922771244
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Monocular 3D scene reconstruction has recently seen significant progress. Powered by the modern neural architectures and large-scale data, recent methods achieve high performance in depth estimation from a single image. Meanwhile, reconstructing and decomposing common scenes into individual 3D objects remains a hard challenge due to the large variety of objects, frequent occlusions and complex object relations. Notably, beyond shape and pose estimation of individual objects, applications in robotics and animation require physically-plausible scene reconstruction where objects obey physical principles of non-penetration and realistic contacts. In this work we advance object-level scene reconstruction along two directions. First, we introduceMessyKitchens, a new dataset with real-world scenes featuring cluttered environments and providing high-fidelity object-level ground truth in terms of 3D object shapes, poses and accurate object contacts. Second, we build on the recent SAM 3D approach for single-object reconstruction and extend it with Multi-Object Decoder (MOD) for joint object-level scene reconstruction. To validate our contributions, we demonstrate MessyKitchens to significantly improve previous datasets in registration accuracy and inter-object penetration. We also compare our multi-object reconstruction approach on three datasets and demonstrate consistent and significant improvements of MOD over the state of the art. Our new benchmark, code and pre-trained models will become publicly available on our project website: https://messykitchens.github.io/.

Related papers

• WorldSculpt: Generating Compositional Worlds from Grounded Videos [50.427204240556854]
  We study the problem of generating a compositional 3D representation of a cluttered scene containing hundreds of objects.<n>This task is challenging in densely cluttered scenes, where objects heavily occlude one another and each view reveals only a fraction of their geometry.<n>We show that complex scenes with hundreds of objects can instead be generated compositionally by adapting a strong single-object 3D generative prior to multi-view observations.
  arXiv  Detail & Related papers  (2026-09-04T17:59:35Z)
• SAM3D-Phys: Towards Multi-Object Interactive Simulation in Real World [42.46212677643043]
  SAM3D-Phys is a framework that integrates scene reconstruction with generative 3D priors of SAM3D to recover physically simulatable objects.<n>By recovering complete object geometry and restoring its pose and appearance within the scene, SAM3D-Phys produces clean object representations suitable for physics-based simulation.
  arXiv  Detail & Related papers  (2026-05-28T17:07:51Z)
• FurnSet: Exploiting Repeats for 3D Scene Reconstruction [3.948390930350905]
  Single-view 3D scene reconstruction involves inferring both object geometry and spatial layout.<n>Existing methods typically reconstruct objects independently or rely on implicit scene context.<n>We propose FurnSet, a framework that explicitly identifies and leverages repeated object instances to improve reconstruction.
  arXiv  Detail & Related papers  (2026-04-22T01:27:13Z)
• SAM 3D: 3Dfy Anything in Images [99.1053358868456]
  We present SAM 3D, a generative model for visually grounded 3D object reconstruction, predicting geometry, texture, and layout from a single image.<n>We achieve this with a human- and model-in-the-loop pipeline for annotating object shape, texture, and pose.<n>We will release our code and model weights, an online demo, and a new challenging benchmark for in-the-wild 3D object reconstruction.
  arXiv  Detail & Related papers  (2025-11-20T18:31:46Z)
• SCORP: Scene-Consistent Object Refinement via Proxy Generation and Tuning [46.441761732998536]
  We introduce Scene-Consistent Object Refinement via Proxy Generation and Tuning (SCORP)<n>SCORP is a novel 3D enhancement framework that leverages 3D generative priors to recover fine-grained object geometry and appearance under missing views.<n>It achieves consistent gains over recent state-of-the-art baselines on both novel view synthesis and geometry completion tasks.
  arXiv  Detail & Related papers  (2025-06-30T13:26:21Z)
• Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations [114.57192386025373]
  Object-X is a versatile multi-modal 3D representation framework.<n>It can encoding rich object embeddings and decoding them back into geometric and visual reconstructions.<n>It supports a range of downstream tasks, including scene alignment, single-image 3D object reconstruction, and localization.
  arXiv  Detail & Related papers  (2025-06-05T09:14:42Z)
• Zero-Shot Multi-Object Scene Completion [59.325611678171974]
  We present a 3D scene completion method that recovers the complete geometry of multiple unseen objects in complex scenes from a single RGB-D image. Our method outperforms the current state-of-the-art on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-03-21T17:59:59Z)
• HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video [70.11702620562889]
  HOLD -- the first category-agnostic method that reconstructs an articulated hand and object jointly from a monocular interaction video. We develop a compositional articulated implicit model that can disentangled 3D hand and object from 2D images. Our method does not rely on 3D hand-object annotations while outperforming fully-supervised baselines in both in-the-lab and challenging in-the-wild settings.
  arXiv  Detail & Related papers  (2023-11-30T10:50:35Z)
• Unsupervised Learning of 3D Object Categories from Videos in the Wild [75.09720013151247]
  We focus on learning a model from multiple views of a large collection of object instances. We propose a new neural network design, called warp-conditioned ray embedding (WCR), which significantly improves reconstruction. Our evaluation demonstrates performance improvements over several deep monocular reconstruction baselines on existing benchmarks.
  arXiv  Detail & Related papers  (2021-03-30T17:57:01Z)
• CoReNet: Coherent 3D scene reconstruction from a single RGB image [43.74240268086773]
  We build on advances in deep learning to reconstruct the shape of a single object given only one RBG image as input. We propose three extensions: (1) ray-traced skip connections that propagate local 2D information to the output 3D volume in a physically correct manner; (2) a hybrid 3D volume representation that enables building translation equivariant models; and (3) a reconstruction loss tailored to capture overall object geometry. We reconstruct all objects jointly in one pass, producing a coherent reconstruction, where all objects live in a single consistent 3D coordinate frame relative to the camera and they do not intersect in 3D space.
  arXiv  Detail & Related papers  (2020-04-27T17:53:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.