Probing the 3D Object-Level Understanding of Pre-Trained Detection Transformers

• URL: http://arxiv.org/abs/2608.01495v1
• Date: Sun, 02 Aug 2026 20:56:14 GMT
• Title: Probing the 3D Object-Level Understanding of Pre-Trained Detection Transformers
• Abstract summary: We investigate what pre-trained 2D detection transformers understand about the 3D properties of objects.<n>Across a range of detection transformer models, our results show a surprisingly strong and previously unknown ability of 2D DETR models to represent useful information.
• Score: 5.05030444084852
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Detection transformer models, including DETR and its extensions, learn to output a set of object-level embeddings that can be simultaneously decoded into 2D bounding boxes and class distributions. In this paper, we investigate what pre-trained 2D detection transformers understand about the 3D properties of objects. Specifically, we investigate the extent to which properties including the depth of objects from the camera and the 3D location of objects relative to the camera can be recovered from object-level embeddings using linear and non-linear probes. Across a range of detection transformer models, our results show a surprisingly strong and previously unknown ability of 2D DETR models to represent useful information about the 3D properties of objects, despite the complete lack of 3D supervision during model pre-training.

Related papers

• 3D-DLP: Self-Supervised 3D Object-Centric Scene Representation Learning [37.932415885988924]
  We introduce 3D-DLP, a self-supervised object-centric representation learning model that decomposes scene-level RGB-D or voxel observations into a set of 3D latent particles.<n>The model learns interpretable per-particle segmentation maps through an end-to-end self-supervised reconstruction objective.<n>We show that leveraging these compact 3D latent particles for downstream robotic manipulation improves performance over baselines.
  arXiv  Detail & Related papers  (2026-06-17T18:00:08Z)
• Gaussian Splatting is an Effective Data Generator for 3D Object Detection [12.882763124978535]
  We use recent advancements in 3D reconstruction based on Gaussian Splatting for 3D object placement in driving scenes.<n>Our approach places 3D objects directly in the reconstructed 3D space with explicitly imposed geometric transformations.<n>This ensures both the physical plausibility of object placement and highly accurate 3D pose and position annotations.
  arXiv  Detail & Related papers  (2025-04-23T14:10:36Z)
• DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation [49.32104127246474]
  DriveGEN is a training-free controllable Text-to-Image Diffusion Generation.<n>It consistently preserves objects with precise 3D geometry across diverse Out-of-Distribution generations.
  arXiv  Detail & Related papers  (2025-03-14T06:35:38Z)
• V-MIND: Building Versatile Monocular Indoor 3D Detector with Diverse 2D Annotations [17.49394091283978]
  V-MIND (Versatile Monocular INdoor Detector) enhances the performance of indoor 3D detectors across a diverse set of object classes.<n>We generate 3D training data by converting large-scale 2D images into 3D point clouds and subsequently deriving pseudo 3D bounding boxes.<n>V-MIND achieves state-of-the-art object detection performance across a wide range of classes on the Omni3D indoor dataset.
  arXiv  Detail & Related papers  (2024-12-16T03:28:00Z)
• VFMM3D: Releasing the Potential of Image by Vision Foundation Model for Monocular 3D Object Detection [80.62052650370416]
  monocular 3D object detection holds significant importance across various applications, including autonomous driving and robotics. In this paper, we present VFMM3D, an innovative framework that leverages the capabilities of Vision Foundation Models (VFMs) to accurately transform single-view images into LiDAR point cloud representations.
  arXiv  Detail & Related papers  (2024-04-15T03:12:12Z)
• Transformer-based stereo-aware 3D object detection from binocular images [82.85433941479216]
  We explore the model design of Transformers in binocular 3D object detection. To achieve this goal, we present TS3D, a Stereo-aware 3D object detector. Our proposed TS3D achieves a 41.29% Moderate Car detection average precision on the KITTI test set and takes 88 ms to detect objects from each binocular image pair.
  arXiv  Detail & Related papers  (2023-04-24T08:29:45Z)
• OA-DET3D: Embedding Object Awareness as a General Plug-in for Multi-Camera 3D Object Detection [77.43427778037203]
  We introduce OA-DET3D, a plug-in module that improves 3D object detection.<n> OA-DET3D boosts the representation of objects by leveraging object-centric depth information and foreground pseudo points.<n>We conduct extensive experiments on the nuScenes dataset and Argoverse 2 dataset to validate the merits of OA-DET3D.
  arXiv  Detail & Related papers  (2023-01-13T06:02:31Z)
• Aerial Monocular 3D Object Detection [67.20369963664314]
  DVDET is proposed to achieve aerial monocular 3D object detection in both the 2D image space and the 3D physical space.<n>To address the severe view deformation issue, we propose a novel trainable geo-deformable transformation module.<n>To encourage more researchers to investigate this area, we will release the dataset and related code.
  arXiv  Detail & Related papers  (2022-08-08T08:32:56Z)
• Homography Loss for Monocular 3D Object Detection [54.04870007473932]
  A differentiable loss function, termed as Homography Loss, is proposed to achieve the goal, which exploits both 2D and 3D information. Our method yields the best performance compared with the other state-of-the-arts by a large margin on KITTI 3D datasets.
  arXiv  Detail & Related papers  (2022-04-02T03:48:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.