KitchenTwin: Semantically and Geometrically Grounded 3D Kitchen Digital Twins

• URL: http://arxiv.org/abs/2603.24684v1
• Date: Wed, 25 Mar 2026 18:03:45 GMT
• Title: KitchenTwin: Semantically and Geometrically Grounded 3D Kitchen Digital Twins
• Abstract summary: Embodied AI training and evaluation require object-centric digital twins with accurate metric geometry and semantic grounding.<n>Recent transformer-based feedforward reconstruction methods can efficiently predict global point clouds from sparse monocular videos.<n>This mismatch prevents the reliable fusion of these dimensionless point cloud predictions with locally reconstructed object meshes.
• Score: 11.881796071022157
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied AI training and evaluation require object-centric digital twin environments with accurate metric geometry and semantic grounding. Recent transformer-based feedforward reconstruction methods can efficiently predict global point clouds from sparse monocular videos, yet these geometries suffer from inherent scale ambiguity and inconsistent coordinate conventions. This mismatch prevents the reliable fusion of these dimensionless point cloud predictions with locally reconstructed object meshes. We propose a novel scale-aware 3D fusion framework that registers visually grounded object meshes with transformer-predicted global point clouds to construct metrically consistent digital twins. Our method introduces a Vision-Language Model (VLM)-guided geometric anchor mechanism that resolves this fundamental coordinate mismatch by recovering an accurate real-world metric scale. To fuse these networks, we propose a geometry-aware registration pipeline that explicitly enforces physical plausibility through gravity-aligned vertical estimation, Manhattan-world structural constraints, and collision-free local refinement. Experiments on real indoor kitchen environments demonstrate improved cross-network object alignment and geometric consistency for downstream tasks, including multi-primitive fitting and metric measurement. We additionally introduce an open-source indoor digital twin dataset with metrically scaled scenes and semantically grounded and registered object-centric mesh annotations.

Related papers

• OmniPoint: Universal Monocular Metric Pointcloud from Any Camera [79.24417597712802]
  We present OmniPoint, a unified framework designed to generalize metric reconstruction across diverse imaging sensors.<n>To overcome projection rigidity, OmniPoint abandons conventional planar depth regression.<n>It instead adopts a decoupled ray and distance representation alongside a decoupled training objective.
  arXiv  Detail & Related papers  (2026-09-08T19:45:58Z)
• Beyond Relative Geometry: Metric-Aware Geometry Perception for Robotics [16.80010144055988]
  Metric-Aware Geometry Perception (MAGP) is an end-to-end, plug-and-play framework for metric geometry reconstruction.<n>At its core, Metric Scale Equivariant Augmentation encourages the model to reconstruct metric geometry from camera parameters and depth observations.<n> Flexible Metric Conditioning enables MAGP to support arbitrary view counts and combinations of camera and depth inputs, improving robustness to heterogeneous robotic sensing configurations.
  arXiv  Detail & Related papers  (2026-08-27T02:07:22Z)
• 3DTMDet: A Dual-Path Synergy Network of Transformer and SSM for 3D Object Detection in Point Clouds [15.3066954300581]
  3DTMDet is a novel detection network that combines state space models (Mamba) with Transformers.<n>We propose the 3D Hybrid Mamba Transformer (3DHMT) block, which uses an SSM-Attention-SSM pipeline to balance global context understanding and local detail preservation.<n>Experiments conducted on the KITTI and ONCE datasets have shown that 3DTMDet outperforms state-of-the-art detectors.
  arXiv  Detail & Related papers  (2026-05-15T02:33:19Z)
• Boosting Cross-Domain Point Classification via Distilling Relational Priors from 2D Transformers [59.0181939916084]
  Traditional 3D networks mainly focus on local geometric details and ignore the topological structure between local geometries. We propose a novel Priors Distillation (RPD) method to extract priors from the well-trained transformers on massive images. Experiments on the PointDA-10 and the Sim-to-Real datasets verify that the proposed method consistently achieves the state-of-the-art performance of UDA for point cloud classification.
  arXiv  Detail & Related papers  (2024-07-26T06:29:09Z)
• Mesh Denoising Transformer [104.5404564075393]
  Mesh denoising is aimed at removing noise from input meshes while preserving their feature structures. SurfaceFormer is a pioneering Transformer-based mesh denoising framework. New representation known as Local Surface Descriptor captures local geometric intricacies. Denoising Transformer module receives the multimodal information and achieves efficient global feature aggregation.
  arXiv  Detail & Related papers  (2024-05-10T15:27:43Z)
• Flattening-Net: Deep Regular 2D Representation for 3D Point Cloud Analysis [66.49788145564004]
  We present an unsupervised deep neural architecture called Flattening-Net to represent irregular 3D point clouds of arbitrary geometry and topology. Our methods perform favorably against the current state-of-the-art competitors.
  arXiv  Detail & Related papers  (2022-12-17T15:05:25Z)
• TerrainMesh: Metric-Semantic Terrain Reconstruction from Aerial Images Using Joint 2D-3D Learning [20.81202315793742]
  This paper develops a joint 2D-3D learning approach to reconstruct a local metric-semantic mesh at each camera maintained by a visual odometry algorithm. The mesh can be assembled into a global environment model to capture the terrain topology and semantics during online operation.
  arXiv  Detail & Related papers  (2022-04-23T05:18:39Z)
• Category-Level Metric Scale Object Shape and Pose Estimation [73.92460712829188]
  We propose a framework that jointly estimates a metric scale shape and pose from a single RGB image. We validated our method on both synthetic and real-world datasets to evaluate category-level object pose and shape.
  arXiv  Detail & Related papers  (2021-09-01T12:16:46Z)
• Learning Geometry-Disentangled Representation for Complementary Understanding of 3D Object Point Cloud [50.56461318879761]
  We propose Geometry-Disentangled Attention Network (GDANet) for 3D image processing. GDANet disentangles point clouds into contour and flat part of 3D objects, respectively denoted by sharp and gentle variation components. Experiments on 3D object classification and segmentation benchmarks demonstrate that GDANet achieves the state-of-the-arts with fewer parameters.
  arXiv  Detail & Related papers  (2020-12-20T13:35:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.