Scene Adaptive Sparse Transformer for Event-based Object Detection

• URL: http://arxiv.org/abs/2404.01882v1
• Date: Tue, 2 Apr 2024 12:15:25 GMT
• Title: Scene Adaptive Sparse Transformer for Event-based Object Detection
• Authors: Yansong Peng, Hebei Li, Yueyi Zhang, Xiaoyan Sun, Feng Wu,
• Abstract summary: We propose a Scene Adaptive Sparse Transformer (SAST) for event-based object detection. SAST enables window-token co-sparsification, significantly enhancing fault tolerance and reducing computational overhead. It outperforms all other dense and sparse networks in both performance and efficiency on two large-scale event-based object detection datasets.
• Score: 40.04162039970849
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While recent Transformer-based approaches have shown impressive performances on event-based object detection tasks, their high computational costs still diminish the low power consumption advantage of event cameras. Image-based works attempt to reduce these costs by introducing sparse Transformers. However, they display inadequate sparsity and adaptability when applied to event-based object detection, since these approaches cannot balance the fine granularity of token-level sparsification and the efficiency of window-based Transformers, leading to reduced performance and efficiency. Furthermore, they lack scene-specific sparsity optimization, resulting in information loss and a lower recall rate. To overcome these limitations, we propose the Scene Adaptive Sparse Transformer (SAST). SAST enables window-token co-sparsification, significantly enhancing fault tolerance and reducing computational overhead. Leveraging the innovative scoring and selection modules, along with the Masked Sparse Window Self-Attention, SAST showcases remarkable scene-aware adaptability: It focuses only on important objects and dynamically optimizes sparsity level according to scene complexity, maintaining a remarkable balance between performance and computational cost. The evaluation results show that SAST outperforms all other dense and sparse networks in both performance and efficiency on two large-scale event-based object detection datasets (1Mpx and Gen1). Code: https://github.com/Peterande/SAST

Related papers

• AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation [29.34754905469359]
  AVESFormer is the first real-time Audio-Visual Efficient transformer that achieves fast, efficient and light-weight simultaneously. AVESFormer significantly enhances model performance, achieving 79.9% on S4, 57.9% on MS3 and 31.2% on AVSS.
  arXiv  Detail & Related papers  (2024-08-03T08:25:26Z)
• Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference [14.030836300221756]
  textbfSparse-Tuning is a novel PEFT method that accounts for the information redundancy in images and videos. Sparse-Tuning minimizes the quantity of tokens processed at each layer, leading to a quadratic reduction in computational and memory overhead. Our results show that our Sparse-Tuning reduces GFLOPs to textbf62%-70% of the original ViT-B while achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-05-23T15:34:53Z)
• Rethinking Efficient and Effective Point-based Networks for Event Camera Classification and Regression: EventMamba [11.400397931501338]
  Event cameras efficiently detect changes in ambient light with low latency and high dynamic range while consuming minimal power. Most current approach to processing event data often involves converting it into frame-based representations. Point Cloud is a popular representation for 3D processing and is better suited to match the sparse and asynchronous nature of the event camera. We propose EventMamba, an efficient and effective Point Cloud framework that achieves competitive results even compared to the state-of-the-art (SOTA) frame-based method.
  arXiv  Detail & Related papers  (2024-05-09T21:47:46Z)
• CageViT: Convolutional Activation Guided Efficient Vision Transformer [90.69578999760206]
  This paper presents an efficient vision Transformer, called CageViT, that is guided by convolutional activation to reduce computation. Our CageViT, unlike current Transformers, utilizes a new encoder to handle the rearranged tokens. Experimental results demonstrate that the proposed CageViT outperforms the most recent state-of-the-art backbones by a large margin in terms of efficiency.
  arXiv  Detail & Related papers  (2023-05-17T03:19:18Z)
• Adaptive Sparse Convolutional Networks with Global Context Enhancement for Faster Object Detection on Drone Images [26.51970603200391]
  This paper investigates optimizing the detection head based on the sparse convolution. It suffers from inadequate integration of contextual information of tiny objects. We propose a novel global context-enhanced adaptive sparse convolutional network.
  arXiv  Detail & Related papers  (2023-03-25T14:42:50Z)
• Skip-Attention: Improving Vision Transformers by Paying Less Attention [55.47058516775423]
  Vision computation transformers (ViTs) use expensive self-attention operations in every layer. We propose SkipAt, a method to reuse self-attention from preceding layers to approximate attention at one or more subsequent layers. We show the effectiveness of our method in image classification and self-supervised learning on ImageNet-1K, semantic segmentation on ADE20K, image denoising on SIDD, and video denoising on DAVIS.
  arXiv  Detail & Related papers  (2023-01-05T18:59:52Z)
• Efficient Decoder-free Object Detection with Transformers [75.00499377197475]
  Vision transformers (ViTs) are changing the landscape of object detection approaches. We propose a decoder-free fully transformer-based (DFFT) object detector. DFFT_SMALL achieves high efficiency in both training and inference stages.
  arXiv  Detail & Related papers  (2022-06-14T13:22:19Z)
• Event Transformer. A sparse-aware solution for efficient event data processing [9.669942356088377]
  Event Transformer (EvT) is a framework that effectively takes advantage of event-data properties to be highly efficient and accurate. EvT is evaluated on different event-based benchmarks for action and gesture recognition. Results show better or comparable accuracy to the state-of-the-art while requiring significantly less computation resources.
  arXiv  Detail & Related papers  (2022-04-07T10:49:17Z)
• SALISA: Saliency-based Input Sampling for Efficient Video Object Detection [58.22508131162269]
  We propose SALISA, a novel non-uniform SALiency-based Input SAmpling technique for video object detection. We show that SALISA significantly improves the detection of small objects.
  arXiv  Detail & Related papers  (2022-04-05T17:59:51Z)
• Patch Slimming for Efficient Vision Transformers [107.21146699082819]
  We study the efficiency problem for visual transformers by excavating redundant calculation in given networks. We present a novel patch slimming approach that discards useless patches in a top-down paradigm. Experimental results on benchmark datasets demonstrate that the proposed method can significantly reduce the computational costs of vision transformers.
  arXiv  Detail & Related papers  (2021-06-05T09:46:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.