Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

• URL: http://arxiv.org/abs/2608.07579v1
• Date: Tue, 04 Aug 2026 21:13:59 GMT
• Title: Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real
• Abstract summary: The AI City Challenge 2026 Track 1 evaluates multi-camera 3D perception in large indoor warehouses under a synthetic-to-real (Sim2Real) setting.<n>We use two RGB-only routes as a controlled test of one hypothesis: that cross-view geometric consistency, not monocular depth accuracy, governs performance under Sim2Real.
• Score: 1.7289819674602296
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The AI City Challenge 2026 Track 1 evaluates multi-camera 3D perception in large indoor warehouses under a synthetic-to-real (Sim2Real) setting; depth is available only for training and validation, so inference is RGB-only. We use two RGB-only routes as a controlled test of one hypothesis: that cross-view geometric consistency, not monocular depth accuracy, governs performance under Sim2Real. The first is a geometry-first pipeline: YOLO11x detection, homography lifting to the world frame, class-level 3D size priors, multi-camera fusion, world-coordinate tracking, and offline tracklet stitching. The second is estimated-depth pseudo-LiDAR: monocular depth (D4RT, Metric3D~v2) back-projected into a fused point cloud and passed to a 3D detector (V-DETR), mirroring prior point-cloud winners that used depth. The gap is decisive: geometry-first reaches 13.0 3D HOTA (51.6 LocA), whereas pseudo-LiDAR collapses to 0.12 (9.2 LocA). We trace the collapse to cross-view inconsistency of monocular depth---scale correction is necessary but not sufficient---which domain-adaptation fine-tuning does not repair within budget. Within the geometry pipeline, offline stitching is the only intervention that helps; SAHI detection, appearance Re-ID, learned lifting, RT-DETR ensembling, test-time augmentation, and domain randomization all fail to beat the baseline detector. The bottlenecks are complementary: detection quality bounds the geometry route (DetA), localization consistency bounds pseudo-LiDAR (LocA). We release a complete, reproducible RGB-only pipeline and ablation.

Related papers

• When Depth Hurts: Reliability-Aware Geometry Distillation for Depth-Free RGB-D Salient Object Detection [39.23633029110609]
  Missing regions, blurred boundaries, and structural artifacts can propagate through multimodal fusion and make an RGB-D detector less accurate than its RGB-only counterpart.<n>We propose method, a reliability-aware geometry distillation framework developed for RGB-D SOD benchmarks without using dataset-provided depth during training or inference.<n>Method achieves the best or tied-best result in 26 of 36 metric-dataset comparisons against ten recent RGB-D SOD methods, including a 13.4% relative MAE reduction on ReDWeb-S.
  arXiv  Detail & Related papers  (2026-09-03T05:29:10Z)
• GeoRay: Gauge-Aware Feed-Forward Satellite 3D Reconstruction in the Geodetic Frame [11.043507429399414]
  Method treats perspective-pretrained features, absolute elevation, and monocular and multi-view cues.<n> lightweight ray-consistent adapters make a frozen backbone matchable along native RPC rays.<n>bench, our absolute-frame benchmark of eighteen systems, scores absolute placement without registration or test-reference leakage.
  arXiv  Detail & Related papers  (2026-08-30T09:27:36Z)
• Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs [62.865932175477035]
  Metric 3D object detection is a core capability for embodied agents, yet most reliable systems lean on depth sensors.<n>We propose Map-Det3D, an online multi-view 3D object detection model that brings detection directly into a 3D space reconstructed from RGB.
  arXiv  Detail & Related papers  (2026-08-12T15:33:42Z)
• RealDenseFace: Real-time Monocular 3D Face Reconstruction from Dense UV-space Priors [28.196602417641927]
  We present RealDenseFace, a real-time optimization-based 3D face reconstruction method with dense UV-space network predictions.<n>Our method achieves state-of-the-art accuracy on the NeRSemble SVFR benchmark.
  arXiv  Detail & Related papers  (2026-08-10T08:04:36Z)
• One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models [27.522658390957886]
  A faithful 3D world representation should account for layered geometry, where a single camera ray may contain multiple valid surfaces.<n>Monocular depth estimation reduces this structure to one scalar depth per pixel.<n> Transparent scenes make this ambiguity measurable, turning the supervised target into a convention of annotation, data, and training rather than a scene-intrinsic truth.<n>We introduce MultiDepth-3k, a sparse benchmark for measuring depth-layer preference and multi-layer spatial relationship accuracy.
  arXiv  Detail & Related papers  (2026-06-28T20:54:18Z)
• Geometry OR Tracker: Universal Geometric Operating Room Tracking [61.399734016038614]
  In operating rooms (OR), world-scale multi-view 3D tracking supports downstream applications such as surgeon behavior recognition.<n>Camera calibration and RGB-D registration are always unreliable, leading to cross-view geometric inconsistency.<n>We introduce Geometry OR Tracker, a two-stage pipeline that rectifies imprecise calibration into a scaleconsistent and geometrically consistent camera setup.
  arXiv  Detail & Related papers  (2026-02-28T09:21:21Z)
• Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation [53.09168514034483]
  Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions.<n>We propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model.<n>Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap.
  arXiv  Detail & Related papers  (2026-02-27T08:54:20Z)
• Systematic Evaluation of Depth Backbones and Semantic Cues for Monocular Pseudo-LiDAR 3D Detection [0.0]
  We evaluate how depth backbones and feature engineering affect a monocular Pseudo-LiDAR pipeline on the KITTI validation split.<n>Under an off-the-shelf LiDAR detector, depth-backbone choice and geometric fidelity dominate performance, outweighing secondary feature injection.
  arXiv  Detail & Related papers  (2026-01-07T05:57:19Z)
• Tri-Perspective View Decomposition for Geometry-Aware Depth Completion [24.98850285904668]
  Tri-Perspective view Decomposition (TPVD) is a novel framework that can explicitly model 3D geometry. TPVD decomposes the original point cloud into three 2D views. TPVD outperforms existing methods on KITTI, NYUv2, and SUN RGBD.
  arXiv  Detail & Related papers  (2024-03-22T07:45:50Z)
• Progressive Coordinate Transforms for Monocular 3D Object Detection [52.00071336733109]
  We propose a novel and lightweight approach, dubbed em Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations. In this paper, we propose a novel and lightweight approach, dubbed em Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations.
  arXiv  Detail & Related papers  (2021-08-12T15:22:33Z)
• Anchor-free 3D Single Stage Detector with Mask-Guided Attention for Point Cloud [79.39041453836793]
  We develop a novel single-stage 3D detector for point clouds in an anchor-free manner. We overcome this by converting the voxel-based sparse 3D feature volumes into the sparse 2D feature maps. We propose an IoU-based detection confidence re-calibration scheme to improve the correlation between the detection confidence score and the accuracy of the bounding box regression.
  arXiv  Detail & Related papers  (2021-08-08T13:42:13Z)
• Learning to Predict the 3D Layout of a Scene [0.3867363075280544]
  We propose a method that only uses a single RGB image, thus enabling applications in devices or vehicles that do not have LiDAR sensors. We use the KITTI dataset for training, which consists of street traffic scenes with class labels, 2D bounding boxes and 3D annotations with seven degrees of freedom. We achieve a mean average precision of 47.3% for moderately difficult data, measured at a 3D intersection over union threshold of 70%, as required by the official KITTI benchmark; outperforming previous state-of-the-art single RGB only methods by a large margin.
  arXiv  Detail & Related papers  (2020-11-19T17:23:30Z)
• End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection [62.34374949726333]
  Pseudo-LiDAR (PL) has led to a drastic reduction in the accuracy gap between methods based on LiDAR sensors and those based on cheap stereo cameras. PL combines state-of-the-art deep neural networks for 3D depth estimation with those for 3D object detection by converting 2D depth map outputs to 3D point cloud inputs. We introduce a new framework based on differentiable Change of Representation (CoR) modules that allow the entire PL pipeline to be trained end-to-end.
  arXiv  Detail & Related papers  (2020-04-07T02:18:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.