Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors

• URL: http://arxiv.org/abs/2604.14563v1
• Date: Thu, 16 Apr 2026 02:46:53 GMT
• Title: Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors
• Abstract summary: We propose SEPatch3D, a novel framework that dynamically adjusts patch sizes while preserving semantic information within coarse patches.<n> Experiments on the nuScenes and Argoverse 2 validation sets show that SEPatch3D achieves up to textbf57% faster inference than the StreamPETR baseline and textbf20% higher efficiency than the state-of-the-art ToC3D-faster.
• Score: 18.684602624559734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Transformer (ViT)-based sparse multi-view 3D object detectors have achieved remarkable accuracy but still suffer from high inference latency due to heavy token processing. To accelerate these models, token compression has been widely explored. However, our revisit of existing strategies, such as token pruning, merging, and patch size enlargement, reveals that they often discard informative background cues, disrupt contextual consistency, and lose fine-grained semantics, negatively affecting 3D detection. To overcome these limitations, we propose SEPatch3D, a novel framework that dynamically adjusts patch sizes while preserving critical semantic information within coarse patches. Specifically, we design Spatiotemporal-aware Patch Size Selection (SPSS) that assigns small patches to scenes containing nearby objects to preserve fine details and large patches to background-dominated scenes to reduce computation cost. To further mitigate potential detail loss, Informative Patch Selection (IPS) selects the informative patches for feature refinement, and Cross-Granularity Feature Enhancement (CGFE) injects fine-grained details into selected coarse patches, enriching semantic features. Experiments on the nuScenes and Argoverse 2 validation sets show that SEPatch3D achieves up to \textbf{57\%} faster inference than the StreamPETR baseline and \textbf{20\%} higher efficiency than the state-of-the-art ToC3D-faster, while preserving comparable detection accuracy. Code is available at https://github.com/Mingqj/SEPatch3D.

Related papers

• Budget-Aware Adaptive Adversarial Patches for Black-Box Object Detection [1.8641315013048299]
  We present a query-efficient, budget-adaptive black-box attack that couples a lightweight placer with NES-style pixel updates.<n>Across YOLOv5, Faster R-CNN, and YOLOS, method achieves strong suppression on CNN-based detectors and substantial suppression on the transformer-based detector.
  arXiv  Detail & Related papers  (2026-06-16T13:38:15Z)
• Back to Point: Exploring Point-Language Models for Zero-Shot 3D Anomaly Detection [13.401003727539125]
  Zero-shot (ZS) 3D anomaly detection is crucial for reliable industrial inspection.<n>Existing approaches render 3D point clouds into 2D images and leverage pre-trained Vision-Language Models (VLMs) for anomaly detection.<n>We propose BTP, a novel framework that effectively aligns 3D point cloud and textual embeddings.
  arXiv  Detail & Related papers  (2026-03-23T02:59:00Z)
• PatchBlock: A Lightweight Defense Against Adversarial Patches for Embedded EdgeAI Devices [5.082257334702858]
  Adrial attacks pose a significant challenge to the reliable deployment of machine learning models in EdgeAI applications.<n>We present PatchBlock, a framework designed to detect and neutralize adversarial patches in images.<n>We show that PatchBlock consistently improves robustness, recovering up to 77% of model accuracy under strong patch attacks.
  arXiv  Detail & Related papers  (2026-01-01T15:04:16Z)
• ChangingGrounding: 3D Visual Grounding in Changing Scenes [92.00984845186679]
  Real-world robots localize objects from natural-language instructions while scenes around them keep changing.<n>Most of the existing 3D visual grounding (3DVG) method still assumes a reconstructed and up-to-date point cloud.<n>We introduce ChangingGrounding, the first benchmark that explicitly measures how well an agent can exploit past observations.
  arXiv  Detail & Related papers  (2025-10-16T17:59:16Z)
• SparseFormer: Detecting Objects in HRW Shots via Sparse Vision Transformer [62.11796778482088]
  We present a novel model-agnostic sparse vision transformer, dubbed SparseFormer, to bridge the gap of object detection between close-up and HRW shots.<n>The proposed SparseFormer selectively uses attentive tokens to scrutinize the sparsely distributed windows that may contain objects.<n> experiments on two HRW benchmarks, PANDA and DOTA-v1.0, demonstrate that the proposed SparseFormer significantly improves detection accuracy (up to 5.8%) and speed (up to 3x) over the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-02-11T03:21:25Z)
• Make Your ViT-based Multi-view 3D Detectors Faster via Token Compression [78.93023152602525]
  Slow inference speed is one of the most crucial concerns for deploying multi-view 3D detectors to tasks with high real-time requirements like autonomous driving. We propose a simple yet effective method called TokenCompression3D (ToC3D) Our method can nearly maintain the performance of recent SOTA with up to 30% inference speedup, and the improvements are consistent after scaling up the ViT and input resolution.
  arXiv  Detail & Related papers  (2024-09-01T06:58:08Z)
• Semi-supervised 3D Object Detection with PatchTeacher and PillarMix [71.4908268136439]
  Current semi-supervised 3D object detection methods typically use a teacher to generate pseudo labels for a student. We propose PatchTeacher, which focuses on partial scene 3D object detection to provide high-quality pseudo labels for the student. We introduce three key techniques, i.e., Patch Normalizer, Quadrant Align, and Fovea Selection, to improve the performance of PatchTeacher.
  arXiv  Detail & Related papers  (2024-07-13T06:58:49Z)
• AGO-Net: Association-Guided 3D Point Cloud Object Detection Network [86.10213302724085]
  We propose a novel 3D detection framework that associates intact features for objects via domain adaptation. We achieve new state-of-the-art performance on the KITTI 3D detection benchmark in both accuracy and speed.
  arXiv  Detail & Related papers  (2022-08-24T16:54:38Z)
• Progressive Coordinate Transforms for Monocular 3D Object Detection [52.00071336733109]
  We propose a novel and lightweight approach, dubbed em Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations. In this paper, we propose a novel and lightweight approach, dubbed em Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations.
  arXiv  Detail & Related papers  (2021-08-12T15:22:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.