Towards 3D Semantic Scene Completion for Autonomous Driving: A Meta-Learning Framework Empowered by Deformable Large-Kernel Attention and Mamba Model

• URL: http://arxiv.org/abs/2411.03672v1
• Date: Wed, 06 Nov 2024 05:11:25 GMT
• Title: Towards 3D Semantic Scene Completion for Autonomous Driving: A Meta-Learning Framework Empowered by Deformable Large-Kernel Attention and Mamba Model
• Authors: Yansong Qu, Zilin Huang, Zihao Sheng, Tiantian Chen, Sikai Chen,
• Abstract summary: We introduce MetaSSC, a novel meta-learning-based framework for semantic scene completion (SSC) Our approach begins with a voxel-based semantic segmentation (SS) pretraining task, aimed at exploring the semantics and geometry of incomplete regions. Using simulated cooperative perception datasets, we supervise the perception training of a single vehicle using aggregated sensor data. This meta-knowledge is then adapted to the target domain through a dual-phase training strategy, enabling efficient deployment.
• Score: 1.6835437621159244
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semantic scene completion (SSC) is essential for achieving comprehensive perception in autonomous driving systems. However, existing SSC methods often overlook the high deployment costs in real-world applications. Traditional architectures, such as 3D Convolutional Neural Networks (3D CNNs) and self-attention mechanisms, face challenges in efficiently capturing long-range dependencies within 3D voxel grids, limiting their effectiveness. To address these issues, we introduce MetaSSC, a novel meta-learning-based framework for SSC that leverages deformable convolution, large-kernel attention, and the Mamba (D-LKA-M) model. Our approach begins with a voxel-based semantic segmentation (SS) pretraining task, aimed at exploring the semantics and geometry of incomplete regions while acquiring transferable meta-knowledge. Using simulated cooperative perception datasets, we supervise the perception training of a single vehicle using aggregated sensor data from multiple nearby connected autonomous vehicles (CAVs), generating richer and more comprehensive labels. This meta-knowledge is then adapted to the target domain through a dual-phase training strategy that does not add extra model parameters, enabling efficient deployment. To further enhance the model's capability in capturing long-sequence relationships within 3D voxel grids, we integrate Mamba blocks with deformable convolution and large-kernel attention into the backbone network. Extensive experiments demonstrate that MetaSSC achieves state-of-the-art performance, significantly outperforming competing models while also reducing deployment costs.

Related papers

• S3MOT: Monocular 3D Object Tracking with Selective State Space Model [3.5047603107971397]
  Multi-object tracking in 3D space is essential for advancing robotics and computer applications. It remains a significant challenge in monocular setups due to the difficulty of mining 3D associations from 2D video streams. We present three innovative techniques to enhance the fusion of heterogeneous cues for monocular 3D MOT.
  arXiv  Detail & Related papers  (2025-04-25T04:45:35Z)
• Data-Driven Object Tracking: Integrating Modular Neural Networks into a Kalman Framework [2.1369549137353805]
  We introduce three Neural Network (NN) models that address key challenges in Multi-Object Tracking (MOT) All three networks are designed to be run in a realtime, embedded environment. Our evaluation, conducted on the public KITTI tracking dataset, demonstrates significant improvements in tracking performance.
  arXiv  Detail & Related papers  (2025-04-03T12:13:38Z)
• DSU-Net:An Improved U-Net Model Based on DINOv2 and SAM2 with Multi-scale Cross-model Feature Enhancement [7.9006143460465355]
  This paper proposes a multi-scale feature collabora-tion framework guided by DINOv2 for SAM2, with core innovations in three aspects. It surpasses existing state-of-the-art meth-ods in downstream tasks such as camouflage target detection and salient ob-ject detection, without requiring costly training processes.
  arXiv  Detail & Related papers  (2025-03-27T06:08:24Z)
• Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learning [10.862065825733243]
  Spiking Neural Networks (SNNs) are brain-inspired computational models. Existing SNN models focus on unimodal processing and lack efficient cross-modal information fusion. We propose a semantic-alignment cross-modal residual learning framework for effective audio-visual integration.
  arXiv  Detail & Related papers  (2025-02-18T03:18:29Z)
• LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving [52.83707400688378]
  LargeAD is a versatile and scalable framework designed for large-scale 3D pretraining across diverse real-world driving datasets. Our framework leverages VFMs to extract semantically rich superpixels from 2D images, which are aligned with LiDAR point clouds to generate high-quality contrastive samples. Our approach delivers significant performance improvements over state-of-the-art methods in both linear probing and fine-tuning tasks for both LiDAR-based segmentation and object detection.
  arXiv  Detail & Related papers  (2025-01-07T18:59:59Z)
• GEAL: Generalizable 3D Affordance Learning with Cross-Modal Consistency [50.11520458252128]
  Existing 3D affordance learning methods struggle with generalization and robustness due to limited annotated data. We propose GEAL, a novel framework designed to enhance the generalization and robustness of 3D affordance learning by leveraging large-scale pre-trained 2D models. GEAL consistently outperforms existing methods across seen and novel object categories, as well as corrupted data.
  arXiv  Detail & Related papers  (2024-12-12T17:59:03Z)
• Unleashing the Potential of Mamba: Boosting a LiDAR 3D Sparse Detector by Using Cross-Model Knowledge Distillation [22.653014803666668]
  We propose a Faster LiDAR 3D object detection framework, called FASD, which implements heterogeneous model distillation by adaptively uniform cross-model voxel features. We aim to distill the transformer's capacity for high-performance sequence modeling into Mamba models with low FLOPs, achieving a significant improvement in accuracy through knowledge transfer. We evaluated the framework on datasets and nuScenes, achieving a 4x reduction in resource consumption and a 1-2% performance improvement over the current SoTA methods.
  arXiv  Detail & Related papers  (2024-09-17T09:30:43Z)
• CoMamba: Real-time Cooperative Perception Unlocked with State Space Models [39.87600356189242]
  CoMamba is a novel cooperative 3D detection framework designed to leverage state-space models for real-time onboard vehicle perception. CoMamba achieves superior performance compared to existing methods while maintaining real-time processing capabilities.
  arXiv  Detail & Related papers  (2024-09-16T20:02:19Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• Unleashing Network Potentials for Semantic Scene Completion [50.95486458217653]
  This paper proposes a novel SSC framework - Adrial Modality Modulation Network (AMMNet) AMMNet introduces two core modules: a cross-modal modulation enabling the interdependence of gradient flows between modalities, and a customized adversarial training scheme leveraging dynamic gradient competition. Extensive experimental results demonstrate that AMMNet outperforms state-of-the-art SSC methods by a large margin.
  arXiv  Detail & Related papers  (2024-03-12T11:48:49Z)
• Cross-Cluster Shifting for Efficient and Effective 3D Object Detection in Autonomous Driving [69.20604395205248]
  We present a new 3D point-based detector model, named Shift-SSD, for precise 3D object detection in autonomous driving. We introduce an intriguing Cross-Cluster Shifting operation to unleash the representation capacity of the point-based detector. We conduct extensive experiments on the KITTI, runtime, and nuScenes datasets, and the results demonstrate the state-of-the-art performance of Shift-SSD.
  arXiv  Detail & Related papers  (2024-03-10T10:36:32Z)
• Double-chain Constraints for 3D Human Pose Estimation in Images and Videos [21.42410292863492]
  Reconstructing 3D poses from 2D poses lacking depth information is challenging due to the complexity and diversity of human motion. We propose a novel model, called Double-chain Graph Convolutional Transformer (DC-GCT), to constrain the pose. We show that DC-GCT achieves state-of-the-art performance on two challenging datasets.
  arXiv  Detail & Related papers  (2023-08-10T02:41:18Z)
• SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Driving [87.8761593366609]
  SSCBench is a benchmark that integrates scenes from widely used automotive datasets. We benchmark models using monocular, trinocular, and cloud input to assess the performance gap. We have unified semantic labels across diverse datasets to simplify cross-domain generalization testing.
  arXiv  Detail & Related papers  (2023-06-15T09:56:33Z)
• HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving [95.42203932627102]
  3D human pose estimation is an emerging technology, which can enable the autonomous vehicle to perceive and understand the subtle and complex behaviors of pedestrians. Our method efficiently makes use of these complementary signals, in a semi-supervised fashion and outperforms existing methods with a large margin. Specifically, we embed LiDAR points into pixel-aligned multi-modal features, which we pass through a sequence of Transformer refinement stages.
  arXiv  Detail & Related papers  (2022-12-15T11:15:14Z)
• 3D Convolutional with Attention for Action Recognition [6.238518976312625]
  Current action recognition methods use computationally expensive models for learning-temporal dependencies of the action. This paper proposes a deep neural network architecture for learning such dependencies consisting of a 3D convolutional layer, fully connected layers and attention layer. The method first learns spatial features and temporal of actions through 3D-CNN, and then the attention temporal mechanism helps the model to locate attention to essential features.
  arXiv  Detail & Related papers  (2022-06-05T15:12:57Z)
• Learnable Online Graph Representations for 3D Multi-Object Tracking [156.58876381318402]
  We propose a unified and learning based approach to the 3D MOT problem. We employ a Neural Message Passing network for data association that is fully trainable. We show the merit of the proposed approach on the publicly available nuScenes dataset by achieving state-of-the-art performance of 65.6% AMOTA and 58% fewer ID-switches.
  arXiv  Detail & Related papers  (2021-04-23T17:59:28Z)
• Enhanced 3D Human Pose Estimation from Videos by using Attention-Based Neural Network with Dilated Convolutions [12.900524511984798]
  We show a systematic design for how conventional networks and other forms of constraints can be incorporated into the attention framework. We achieve this by adapting temporal receptive field via a multi-scale structure of dilated convolutions. Our method achieves the state-of-the-art performance and outperforms existing methods by reducing the mean per joint position error to 33.4 mm on Human3.6M dataset.
  arXiv  Detail & Related papers  (2021-03-04T17:26:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.