FOMO-3D: Using Vision Foundation Models for Long-Tailed 3D Object Detection

• URL: http://arxiv.org/abs/2603.08611v1
• Date: Mon, 09 Mar 2026 16:57:23 GMT
• Title: FOMO-3D: Using Vision Foundation Models for Long-Tailed 3D Object Detection
• Abstract summary: Many safety-critical objects appear infrequently in nominal traffic conditions, leading to a severe shortage of training examples from driving data alone.<n>Recent vision foundation models, which are trained on a large corpus of data, can serve as a good source of external prior knowledge.<n>We propose FOMO-3D, the first multi-modal 3D detector to leverage vision foundation models for long-tailed 3D detection.
• Score: 26.468026465568396
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In order to navigate complex traffic environments, self-driving vehicles must recognize many semantic classes pertaining to vulnerable road users or traffic control devices. However, many safety-critical objects (e.g., construction worker) appear infrequently in nominal traffic conditions, leading to a severe shortage of training examples from driving data alone. Recent vision foundation models, which are trained on a large corpus of data, can serve as a good source of external prior knowledge to improve generalization. We propose FOMO-3D, the first multi-modal 3D detector to leverage vision foundation models for long-tailed 3D detection. Specifically, FOMO-3D exploits rich semantic and depth priors from OWLv2 and Metric3Dv2 within a two-stage detection paradigm that first generates proposals with a LiDAR-based branch and a novel camera-based branch, and refines them with attention especially to image features from OWL. Evaluations on real-world driving data show that using rich priors from vision foundation models with careful multi-modal fusion designs leads to large gains for long-tailed 3D detection. Project website is at https://waabi.ai/fomo3d/.

Related papers

• Evaluating 2D and 3D-Aware Vision Foundation Models for Vehicle Attribute Recognition [38.85215563737551]
  This paper presents an empirical benchmark of 14 state-of-the-art 2D and 3D-aware vision foundation models.<n>We evaluate these models as frozen feature extractors via linear probing for vehicle type, make, and model recognition.<n>Our results show that standard 2D self-supervised models, particularly DINOv3, substantially outperform 3D-aware models in fine-grained tasks.
  arXiv  Detail & Related papers  (2026-08-30T17:51:44Z)
• Map-Det3D: Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs [62.865932175477035]
  Metric 3D object detection is a core capability for embodied agents, yet most reliable systems lean on depth sensors.<n>We propose Map-Det3D, an online multi-view 3D object detection model that brings detection directly into a 3D space reconstructed from RGB.
  arXiv  Detail & Related papers  (2026-08-12T15:33:42Z)
• HeightFormer: A Semantic Alignment Monocular 3D Object Detection Method from Roadside Perspective [11.841338298700421]
  We propose a novel 3D object detection framework integrating Spatial Former and Voxel Pooling Former to enhance 2D-to-3D projection based on height estimation. Experiments were conducted using the Rope3D and DAIR-V2X-I dataset, and the results demonstrated the outperformance of the proposed algorithm in the detection of both vehicles and cyclists.
  arXiv  Detail & Related papers  (2024-10-10T09:37:33Z)
• 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features [70.50665869806188]
  3DiffTection is a state-of-the-art method for 3D object detection from single images. We fine-tune a diffusion model to perform novel view synthesis conditioned on a single image. We further train the model on target data with detection supervision.
  arXiv  Detail & Related papers  (2023-11-07T23:46:41Z)
• Generalized Few-Shot 3D Object Detection of LiDAR Point Cloud for Autonomous Driving [91.39625612027386]
  We propose a novel task, called generalized few-shot 3D object detection, where we have a large amount of training data for common (base) objects, but only a few data for rare (novel) classes. Specifically, we analyze in-depth differences between images and point clouds, and then present a practical principle for the few-shot setting in the 3D LiDAR dataset. To solve this task, we propose an incremental fine-tuning method to extend existing 3D detection models to recognize both common and rare objects.
  arXiv  Detail & Related papers  (2023-02-08T07:11:36Z)
• HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving [95.42203932627102]
  3D human pose estimation is an emerging technology, which can enable the autonomous vehicle to perceive and understand the subtle and complex behaviors of pedestrians. Our method efficiently makes use of these complementary signals, in a semi-supervised fashion and outperforms existing methods with a large margin. Specifically, we embed LiDAR points into pixel-aligned multi-modal features, which we pass through a sequence of Transformer refinement stages.
  arXiv  Detail & Related papers  (2022-12-15T11:15:14Z)
• BEV-MAE: Bird's Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios [51.285561119993105]
  We present BEV-MAE, an efficient masked autoencoder pre-training framework for LiDAR-based 3D object detection in autonomous driving. Specifically, we propose a bird's eye view (BEV) guided masking strategy to guide the 3D encoder learning feature representation. We introduce a learnable point token to maintain a consistent receptive field size of the 3D encoder.
  arXiv  Detail & Related papers  (2022-12-12T08:15:03Z)
• Monocular Quasi-Dense 3D Object Tracking [99.51683944057191]
  A reliable and accurate 3D tracking framework is essential for predicting future locations of surrounding objects and planning the observer's actions in numerous applications such as autonomous driving. We propose a framework that can effectively associate moving objects over time and estimate their full 3D bounding box information from a sequence of 2D images captured on a moving platform.
  arXiv  Detail & Related papers  (2021-03-12T15:30:02Z)
• Ground-aware Monocular 3D Object Detection for Autonomous Driving [6.5702792909006735]
  Estimating the 3D position and orientation of objects in the environment with a single RGB camera is a challenging task for low-cost urban autonomous driving and mobile robots. Most of the existing algorithms are based on the geometric constraints in 2D-3D correspondence, which stems from generic 6D object pose estimation. We introduce a novel neural network module to fully utilize such application-specific priors in the framework of deep learning.
  arXiv  Detail & Related papers  (2021-02-01T08:18:24Z)
• Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation [77.60050239225086]
  We propose an effective training data generation process by fitting a 3D car model with dynamic parts to vehicles in real images. Our approach is fully automatic without any human interaction. We present a multi-task network for VUS parsing and a multi-stream network for VHI parsing.
  arXiv  Detail & Related papers  (2020-12-15T03:03:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.