Efficient Multi-View 3D Object Detection by Dynamic Token Selection and Fine-Tuning

• URL: http://arxiv.org/abs/2604.13586v1
• Date: Wed, 15 Apr 2026 07:44:01 GMT
• Title: Efficient Multi-View 3D Object Detection by Dynamic Token Selection and Fine-Tuning
• Abstract summary: Existing multi-view 3D object detection approaches widely adopt large-scale pre-trained vision transformer (ViT)-based foundation models as backbones.<n>We propose an image token compensator combined with a token selection for ViT backbones to accelerate multi-view 3D object detection.
• Score: 16.27181704006191
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Existing multi-view three-dimensional (3D) object detection approaches widely adopt large-scale pre-trained vision transformer (ViT)-based foundation models as backbones, being computationally complex. To address this problem, current state-of-the-art (SOTA) \texttt{ToC3D} for efficient multi-view ViT-based 3D object detection employs ego-motion-based relevant token selection. However, there are two key limitations: (1) The fixed layer-individual token selection ratios limit computational efficiency during both training and inference. (2) Full end-to-end retraining of the ViT backbone is required for the multi-view 3D object detection method. In this work, we propose an image token compensator combined with a token selection for ViT backbones to accelerate multi-view 3D object detection. Unlike \texttt{ToC3D}, our approach enables dynamic layer-wise token selection within the ViT backbone. Furthermore, we introduce a parameter-efficient fine-tuning strategy, which trains only the proposed modules, thereby reducing the number of fine-tuned parameters from more than $300$ million (M) to only $1.6$ M. Experiments on the large-scale NuScenes dataset across three multi-view 3D object detection approaches demonstrate that our proposed method decreases computational complexity (GFLOPs) by $48\%$ ... $55\%$, inference latency (on an \texttt{NVIDIA-GV100} GPU) by $9\%$ ... $25\%$, while still improving mean average precision by $1.0\%$ ... $2.8\%$ absolute and NuScenes detection score by $0.4\%$ ... $1.2\%$ absolute compared to so-far SOTA \texttt{ToC3D}.

Related papers

• SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection [2.148411555253816]
  Vision Transformers (ViTs) enable strong multi-view 3D detection but are limited by high inference latency from dense token and query processing across multiple views and large 3D regions.<n>We introduce SToRe3D, a relevance-aligned sparsity framework that jointly selects 2D image tokens and 3D object queries while storing filtered features for reactivation.
  arXiv  Detail & Related papers  (2026-05-13T20:53:03Z)
• H$_{2}$OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers [124.11648300910444]
  We present a hierarchical plug-and-play pruning-and-$-recovering framework, called Hierarchical Hourglass Tokenizer (H$_2$OT)<n>Our method is general-purpose: it can be easily incorporated into common VPT models on both seq2seq and seq2frame pipelines.
  arXiv  Detail & Related papers  (2025-09-08T17:59:59Z)
• Make Your ViT-based Multi-view 3D Detectors Faster via Token Compression [78.93023152602525]
  Slow inference speed is one of the most crucial concerns for deploying multi-view 3D detectors to tasks with high real-time requirements like autonomous driving. We propose a simple yet effective method called TokenCompression3D (ToC3D) Our method can nearly maintain the performance of recent SOTA with up to 30% inference speedup, and the improvements are consistent after scaling up the ViT and input resolution.
  arXiv  Detail & Related papers  (2024-09-01T06:58:08Z)
• PoIFusion: Multi-Modal 3D Object Detection via Fusion at Points of Interest [65.48057241587398]
  PoIFusion is a framework to fuse information of RGB images and LiDAR point clouds at the points of interest (PoIs) Our approach maintains the view of each modality and obtains multi-modal features by computation-friendly projection and computation. We conducted extensive experiments on nuScenes and Argoverse2 datasets to evaluate our approach.
  arXiv  Detail & Related papers  (2024-03-14T09:28:12Z)
• S$^3$-MonoDETR: Supervised Shape&Scale-perceptive Deformable Transformer for Monocular 3D Object Detection [21.96072831561483]
  This paper proposes a novel Supervised Shape&Scale-perceptive Deformable Attention'' (S$3$-DA) module for monocular 3D object detection. Benefiting from this, S$3$-DA effectively estimates receptive fields for query points belonging to any category, enabling them to generate robust query features. Experiments on KITTI and Open datasets demonstrate that S$3$-DA significantly improves the detection accuracy.
  arXiv  Detail & Related papers  (2023-09-02T12:36:38Z)
• MVTN: Learning Multi-View Transformations for 3D Understanding [60.15214023270087]
  We introduce the Multi-View Transformation Network (MVTN), which uses differentiable rendering to determine optimal view-points for 3D shape recognition. MVTN can be trained end-to-end with any multi-view network for 3D shape recognition. Our approach demonstrates state-of-the-art performance in 3D classification and shape retrieval on several benchmarks.
  arXiv  Detail & Related papers  (2022-12-27T12:09:16Z)
• SRCN3D: Sparse R-CNN 3D for Compact Convolutional Multi-View 3D Object Detection and Tracking [12.285423418301683]
  This paper proposes Sparse R-CNN 3D (SRCN3D), a novel two-stage fully-sparse detector that incorporates sparse queries, sparse attention with box-wise sampling, and sparse prediction. Experiments on nuScenes dataset demonstrate that SRCN3D achieves competitive performance in both 3D object detection and multi-object tracking tasks.
  arXiv  Detail & Related papers  (2022-06-29T07:58:39Z)
• VPIT: Real-time Embedded Single Object 3D Tracking Using Voxel Pseudo Images [90.60881721134656]
  We propose a novel voxel-based 3D single object tracking (3D SOT) method called Voxel Pseudo Image Tracking (VPIT) Experiments on KITTI Tracking dataset show that VPIT is the fastest 3D SOT method and maintains competitive Success and Precision values.
  arXiv  Detail & Related papers  (2022-06-06T14:02:06Z)
• Reinforced Axial Refinement Network for Monocular 3D Object Detection [160.34246529816085]
  Monocular 3D object detection aims to extract the 3D position and properties of objects from a 2D input image. Conventional approaches sample 3D bounding boxes from the space and infer the relationship between the target object and each of them, however, the probability of effective samples is relatively small in the 3D space. We propose to start with an initial prediction and refine it gradually towards the ground truth, with only one 3d parameter changed in each step. This requires designing a policy which gets a reward after several steps, and thus we adopt reinforcement learning to optimize it.
  arXiv  Detail & Related papers  (2020-08-31T17:10:48Z)
• Light3DPose: Real-time Multi-Person 3D PoseEstimation from Multiple Views [5.510992382274774]
  We present an approach to perform 3D pose estimation of multiple people from a few calibrated camera views. Our architecture aggregates feature-maps from a 2D pose estimator backbone into a comprehensive representation of the 3D scene. The proposed method is inherently efficient: as a pure bottom-up approach, it is computationally independent of the number of people in the scene.
  arXiv  Detail & Related papers  (2020-04-06T14:12:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.