LLM-Empowered Multimodal Fusion Framework for Autonomous Driving: Semantic Enhancement and Channel-Adaptive Design

• URL: http://arxiv.org/abs/2607.01772v1
• Date: Thu, 02 Jul 2026 06:44:02 GMT
• Title: LLM-Empowered Multimodal Fusion Framework for Autonomous Driving: Semantic Enhancement and Channel-Adaptive Design
• Abstract summary: We propose a Large Language Model-centric Semantic-layer Channel-aware Integrated Perception framework.<n>It fuses a local visual stream with a quality-varying external radar stream used to cover perception-blind spots.<n>Experiments on nuScenes and VIRAT validate our approach.
• Score: 56.76240343283953
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-radar fusion is central to robust autonomous driving, combining dense visual semantics with precise range and velocity measurements from radar. However, real-world fusion quality is fundamentally challenged by dynamically varying input quality, stemming from occlusion, adverse weather, and channel noise. To address this, we re-frame the problem from static data fusion to channel-aware semantic reasoning and propose a Large Language Model-centric Semantic-layer Channel-aware Integrated Perception (LM-SCIP) framework. It places a Large Language Model (LLM) as a central reasoning core to fuse a local visual stream with a quality-varying external radar stream used to cover perception-blind spots. Concretely, LM-SCIP couples a hierarchical radar-vision encoder with a Channel-Adaptive Semantic Module (CASM) that maps link indicators into a "Channel Prompt" to dynamically gate external radar features. A parameter-efficient, LoRA-tuned LLM, in conjunction with a heterogeneous Mixture-of-Experts (H-MoE), then arbitrates between local visual cues and the channel-conditioned radar context. Finally, a decoupled multi-task decoder outputs localization, trajectory forecasting, and image reconstruction. Experiments on nuScenes and VIRAT validate our approach. On nuScenes, under a controlled toggle of radar input, LM-SCIP reduces localization RMSE by 40.0% versus a vision-only baseline. On VIRAT, the model attains a 0.214m localization RMSE and 0.179m minFDE (k=1). These results reveal that the proposed LM-SCIP enables a robust vision-dominant fallback at low SNR and synergistic fusion at high SNR.

Related papers

• RAVEN: Radar Adaptive Vision Encoders for Efficient Chirp-wise Object Detection and Segmentation [11.345174599707859]
  RAVEN is a computationally efficient deep learning architecture for FMCW radar perception.<n>Across automotive radar benchmarks, the approach reports strong object detection and BEV free-space segmentation performance.
  arXiv  Detail & Related papers  (2026-04-06T07:30:45Z)
• MFC-RFNet: A Multi-scale Guided Rectified Flow Network for Radar Sequence Prediction [7.015114232190396]
  Accurate high-resolution precipitation nowcasting from radar echo sequences is crucial for disaster mitigation and economic planning.<n>Key difficulties include modeling complex multi-scale evolution, inter-frame feature misalignment caused by displacement, and efficiently capturing long-range context.<n>We present the Multi-scale Feature Communication Rectified Flow Network (MFRF-Net), a generative framework that integrates multi-scale communication with guided feature fusion.
  arXiv  Detail & Related papers  (2026-01-07T06:24:26Z)
• Unsupervised Radio Map Construction in Mixed LoS/NLoS Indoor Environments [34.91945910235526]
  This paper aims to recover the data collection trajectory directly from the channel propagation sequence.<n>The proposed method achieves an average localization accuracy of 0.65 meters in an indoor environment.
  arXiv  Detail & Related papers  (2025-10-09T09:53:24Z)
• NOVA: Navigation via Object-Centric Visual Autonomy for High-Speed Target Tracking in Unstructured GPS-Denied Environments [56.35569661650558]
  We introduce NOVA, a fully onboard, object-centric framework that enables robust target tracking and collision-aware navigation.<n>Rather than constructing a global map, NOVA formulates perception, estimation, and control entirely in the target's reference frame.<n>We validate NOVA across challenging real-world scenarios, including urban mazes, forest trails, and repeated transitions through buildings with intermittent GPS loss.
  arXiv  Detail & Related papers  (2025-06-23T14:28:30Z)
• Large Language Model-Driven Distributed Integrated Multimodal Sensing and Semantic Communications [5.646293779615063]
  We propose a novel large language model (LLM)-driven distributed integrated multimodal sensing and semantic communication framework.<n>Specifically, our system consists of multiple collaborative sensing devices equipped with RF and camera modules.<n> evaluations on a synthetic multi-view RF-visual dataset generated by the Genesis simulation engine show that LLM-DiSAC achieves a good performance.
  arXiv  Detail & Related papers  (2025-05-20T08:00:00Z)
• Multi-Modality Driven LoRA for Adverse Condition Depth Estimation [61.525312117638116]
  We propose Multi-Modality Driven LoRA (MMD-LoRA) for Adverse Condition Depth Estimation.<n>It consists of two core components: Prompt Driven Domain Alignment (PDDA) and Visual-Text Consistent Contrastive Learning (VTCCL)<n>It achieves state-of-the-art performance on the nuScenes and Oxford RobotCar datasets.
  arXiv  Detail & Related papers  (2024-12-28T14:23:58Z)
• ALSS-YOLO: An Adaptive Lightweight Channel Split and Shuffling Network for TIR Wildlife Detection in UAV Imagery [7.35275281315994]
  Unmanned aerial vehicles (UAVs) equipped with thermal infrared (TIR) cameras play a crucial role in combating nocturnal wildlife poaching. Current traditional lightweight networks deployed on UAVs struggle to extract features from blurry small targets. We develop ALSS-YOLO, an efficient and lightweight detector optimized for TIR aerial images.
  arXiv  Detail & Related papers  (2024-09-10T07:02:01Z)
• Deep Reinforcement Learning for IRS Phase Shift Design in Spatiotemporally Correlated Environments [93.30657979626858]
  We propose a deep actor-critic algorithm that accounts for channel correlations and destination motion. We show that, when channels aretemporally correlated, the inclusion of the SNR in the state representation with function approximation in ways that inhibit convergence.
  arXiv  Detail & Related papers  (2022-11-02T22:07:36Z)
• Distributional Reinforcement Learning for mmWave Communications with Intelligent Reflectors on a UAV [119.97450366894718]
  A novel communication framework that uses an unmanned aerial vehicle (UAV)-carried intelligent reflector (IR) is proposed. In order to maximize the downlink sum-rate, the optimal precoding matrix (at the base station) and reflection coefficient (at the IR) are jointly derived.
  arXiv  Detail & Related papers  (2020-11-03T16:50:37Z)
• Deep Denoising Neural Network Assisted Compressive Channel Estimation for mmWave Intelligent Reflecting Surfaces [99.34306447202546]
  This paper proposes a deep denoising neural network assisted compressive channel estimation for mmWave IRS systems. We first introduce a hybrid passive/active IRS architecture, where very few receive chains are employed to estimate the uplink user-to-IRS channels. The complete channel matrix can be reconstructed from the limited measurements based on compressive sensing.
  arXiv  Detail & Related papers  (2020-06-03T12:18:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.