Learning Content-Aware Multi-Modal Joint Input Pruning via Bird's-Eye-View Representation

• URL: http://arxiv.org/abs/2410.07268v1
• Date: Wed, 9 Oct 2024 03:30:00 GMT
• Title: Learning Content-Aware Multi-Modal Joint Input Pruning via Bird's-Eye-View Representation
• Authors: Yuxin Li, Yiheng Li, Xulei Yang, Mengying Yu, Zihang Huang, Xiaojun Wu, Chai Kiat Yeo,
• Abstract summary: We introduce a novel content-aware multi-modal joint input pruning technique. We validatethe efficacy of our approach through extensive experiments on the NuScenes dataset.
• Score: 11.074747442071729
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In the landscape of autonomous driving, Bird's-Eye-View (BEV) representation has recently garnered substantial academic attention, serving as a transformative framework for the fusion of multi-modal sensor inputs. This BEV paradigm effectively shifts the sensor fusion challenge from a rule-based methodology to a data-centric approach, thereby facilitating more nuanced feature extraction from an array of heterogeneous sensors. Notwithstanding its evident merits, the computational overhead associated with BEV-based techniques often mandates high-capacity hardware infrastructures, thus posing challenges for practical, real-world implementations. To mitigate this limitation, we introduce a novel content-aware multi-modal joint input pruning technique. Our method leverages BEV as a shared anchor to algorithmically identify and eliminate non-essential sensor regions prior to their introduction into the perception model's backbone. We validatethe efficacy of our approach through extensive experiments on the NuScenes dataset, demonstrating substantial computational efficiency without sacrificing perception accuracy. To the best of our knowledge, this work represents the first attempt to alleviate the computational burden from the input pruning point.

Related papers

• BEVPose: Unveiling Scene Semantics through Pose-Guided Multi-Modal BEV Alignment [8.098296280937518]
  We present BEVPose, a framework that integrates BEV representations from camera and lidar data, using sensor pose as a guiding supervisory signal. By leveraging pose information, we align and fuse multi-modal sensory inputs, facilitating the learning of latent BEV embeddings that capture both geometric and semantic aspects of the environment.
  arXiv  Detail & Related papers  (2024-10-28T12:40:27Z)
• QuadBEV: An Efficient Quadruple-Task Perception Framework via Bird's-Eye-View Representation [11.074747442071729]
  Bird's-Eye-View (BEV) perception has become a vital component of autonomous driving systems due to its ability to integrate multiple sensor inputs into a unified representation. We propose QuadBEV, an efficient multitask perception framework that leverages the shared spatial and contextual information across four key tasks. We present comprehensive experiments that validate the effectiveness and robustness of QuadBEV, demonstrating its suitability for real-world applications.
  arXiv  Detail & Related papers  (2024-10-09T03:31:45Z)
• Hierarchical End-to-End Autonomous Driving: Integrating BEV Perception with Deep Reinforcement Learning [23.21761407287525]
  End-to-end autonomous driving offers a streamlined alternative to the traditional modular pipeline. Deep Reinforcement Learning (DRL) has recently gained traction in this domain. We bridge this gap by mapping the DRL feature extraction network directly to the perception phase.
  arXiv  Detail & Related papers  (2024-09-26T09:14:16Z)
• Hierarchical and Decoupled BEV Perception Learning Framework for Autonomous Driving [52.808273563372126]
  This paper proposes a novel hierarchical BEV perception paradigm, aiming to provide a library of fundamental perception modules and user-friendly graphical interface. We conduct the Pretrain-Finetune strategy to effectively utilize large scale public datasets and streamline development processes. We also present a Multi-Module Learning (MML) approach, enhancing performance through synergistic and iterative training of multiple models.
  arXiv  Detail & Related papers  (2024-07-17T11:17:20Z)
• Diffusion-Based Particle-DETR for BEV Perception [94.88305708174796]
  Bird-Eye-View (BEV) is one of the most widely-used scene representations for visual perception in Autonomous Vehicles (AVs) Recent diffusion-based methods offer a promising approach to uncertainty modeling for visual perception but fail to effectively detect small objects in the large coverage of the BEV. Here, we address this problem by combining the diffusion paradigm with current state-of-the-art 3D object detectors in BEV.
  arXiv  Detail & Related papers  (2023-12-18T09:52:14Z)
• Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
  Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust. Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model. We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
  arXiv  Detail & Related papers  (2023-07-31T10:22:33Z)
• Geometric-aware Pretraining for Vision-centric 3D Object Detection [77.7979088689944]
  We propose a novel geometric-aware pretraining framework called GAPretrain. GAPretrain serves as a plug-and-play solution that can be flexibly applied to multiple state-of-the-art detectors. We achieve 46.2 mAP and 55.5 NDS on the nuScenes val set using the BEVFormer method, with a gain of 2.7 and 2.1 points, respectively.
  arXiv  Detail & Related papers  (2023-04-06T14:33:05Z)
• CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning [56.20123080771364]
  We develop a model-agnostic Contrastive-Curiosity-Driven Learning Framework (CCLF) for reinforcement learning. CCLF fully exploit sample importance and improve learning efficiency in a self-supervised manner. We evaluate this approach on the DeepMind Control Suite, Atari, and MiniGrid benchmarks.
  arXiv  Detail & Related papers  (2022-05-02T14:42:05Z)
• Cross-modal Knowledge Distillation for Vision-to-Sensor Action Recognition [12.682984063354748]
  This study introduces an end-to-end Vision-to-Sensor Knowledge Distillation (VSKD) framework. In this VSKD framework, only time-series data, i.e., accelerometer data, is needed from wearable devices during the testing phase. This framework will not only reduce the computational demands on edge devices, but also produce a learning model that closely matches the performance of the computational expensive multi-modal approach.
  arXiv  Detail & Related papers  (2021-10-08T15:06:38Z)
• Feeling of Presence Maximization: mmWave-Enabled Virtual Reality Meets Deep Reinforcement Learning [76.46530937296066]
  This paper investigates the problem of providing ultra-reliable and energy-efficient virtual reality (VR) experiences for wireless mobile users. To ensure reliable ultra-high-definition (UHD) video frame delivery to mobile users, a coordinated multipoint (CoMP) transmission technique and millimeter wave (mmWave) communications are exploited.
  arXiv  Detail & Related papers  (2021-06-03T08:35:10Z)
• Learning Multiplicative Interactions with Bayesian Neural Networks for Visual-Inertial Odometry [44.209301916028124]
  This paper presents an end-to-end multi-modal learning approach for Visual-Inertial Odometry (VIO) It is specifically designed to exploit sensor complementarity in the light of sensor degradation scenarios.
  arXiv  Detail & Related papers  (2020-07-15T11:39:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.