SIMAC: A Semantic-Driven Integrated Multimodal Sensing And Communication Framework

• URL: http://arxiv.org/abs/2503.08726v1
• Date: Tue, 11 Mar 2025 01:04:42 GMT
• Title: SIMAC: A Semantic-Driven Integrated Multimodal Sensing And Communication Framework
• Authors: Yubo Peng, Luping Xiang, Kun Yang, Feibo Jiang, Kezhi Wang, Dapeng Oliver Wu,
• Abstract summary: Single-modality sensing faces limitations in accuracy and capability, and its decoupled implementation with communication systems increases latency.<n>We propose a semantic-driven integrated multimodal sensing and communication framework to overcome these challenges.
• Score: 22.924064428134507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional single-modality sensing faces limitations in accuracy and capability, and its decoupled implementation with communication systems increases latency in bandwidth-constrained environments. Additionally, single-task-oriented sensing systems fail to address users' diverse demands. To overcome these challenges, we propose a semantic-driven integrated multimodal sensing and communication (SIMAC) framework. This framework leverages a joint source-channel coding architecture to achieve simultaneous sensing decoding and transmission of sensing results. Specifically, SIMAC first introduces a multimodal semantic fusion (MSF) network, which employs two extractors to extract semantic information from radar signals and images, respectively. MSF then applies cross-attention mechanisms to fuse these unimodal features and generate multimodal semantic representations. Secondly, we present a large language model (LLM)-based semantic encoder (LSE), where relevant communication parameters and multimodal semantics are mapped into a unified latent space and input to the LLM, enabling channel-adaptive semantic encoding. Thirdly, a task-oriented sensing semantic decoder (SSD) is proposed, in which different decoded heads are designed according to the specific needs of tasks. Simultaneously, a multi-task learning strategy is introduced to train the SIMAC framework, achieving diverse sensing services. Finally, experimental simulations demonstrate that the proposed framework achieves diverse sensing services and higher accuracy.

Related papers

• Modeling and Performance Analysis for Semantic Communications Based on Empirical Results [53.805458017074294]
  We propose an Alpha-Beta-Gamma (ABG) formula to model the relationship between the end-to-end measurement and SNR. For image reconstruction tasks, the proposed ABG formula can well fit the commonly used DL networks, such as SCUNet, and Vision Transformer. To the best of our knowledge, this is the first theoretical expression between end-to-end performance metrics and SNR for semantic communications.
  arXiv  Detail & Related papers  (2025-04-29T06:07:50Z)
• Take What You Need: Flexible Multi-Task Semantic Communications with Channel Adaptation [51.53221300103261]
  This article introduces a novel channel-adaptive and multi-task-aware semantic communication framework based on a masked auto-encoder architecture.<n>A channel-aware extractor is employed to dynamically select relevant information in response to real-time channel conditions.<n> Experimental results demonstrate the superior performance of our framework compared to conventional methods in tasks such as image reconstruction and object detection.
  arXiv  Detail & Related papers  (2025-02-12T09:01:25Z)
• Agent-driven Generative Semantic Communication with Cross-Modality and Prediction [57.335922373309074]
  We propose a novel agent-driven generative semantic communication framework based on reinforcement learning. In this work, we develop an agent-assisted semantic encoder with cross-modality capability, which can track the semantic changes, channel condition, to perform adaptive semantic extraction and sampling. The effectiveness of the designed models has been verified using the UA-DETRAC dataset, demonstrating the performance gains of the overall A-GSC framework.
  arXiv  Detail & Related papers  (2024-04-10T13:24:27Z)
• Large AI Model Empowered Multimodal Semantic Communications [48.73159237649128]
  We propose a Large AI Model-based Multimodal SC (LAMMSC) framework. We first present the Conditional-based Multimodal Alignment (MMA) that enables the transformation between multimodal and unimodal data. Then, a personalized LLM-based Knowledge Base (LKB) is proposed, which allows users to perform personalized semantic extraction or recovery. Finally, we apply the Generative adversarial network-based channel Estimation (CGE) for estimating the wireless channel state information.
  arXiv  Detail & Related papers  (2023-09-03T19:24:34Z)
• Communication-Efficient Framework for Distributed Image Semantic Wireless Transmission [68.69108124451263]
  Federated learning-based semantic communication (FLSC) framework for multi-task distributed image transmission with IoT devices. Each link is composed of a hierarchical vision transformer (HVT)-based extractor and a task-adaptive translator. Channel state information-based multiple-input multiple-output transmission module designed to combat channel fading and noise.
  arXiv  Detail & Related papers  (2023-08-07T16:32:14Z)
• Rate-Adaptive Coding Mechanism for Semantic Communications With Multi-Modal Data [23.597759255020296]
  We propose a distributed multi-modal semantic communication framework incorporating the conventional channel encoder/decoder. We establish a general rate-adaptive coding mechanism for various types of multi-modal semantic tasks. Numerical results show that the proposed mechanism fares better than both conventional communication and existing semantic communication systems.
  arXiv  Detail & Related papers  (2023-05-18T07:31:37Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)
• Non-Orthogonal Multiple Access Enhanced Multi-User Semantic Communication [7.231199109999097]
  Non-orthogonal multiple access (NOMA)-based multi-user semantic communication system named NOMASC is proposed in this paper. System can support semantic tranmission of multiple users with diverse modalities of source information.
  arXiv  Detail & Related papers  (2023-03-12T07:32:42Z)
• Enabling the Wireless Metaverse via Semantic Multiverse Communication [82.47169682083806]
  Metaverse over wireless networks is an emerging use case of the sixth generation (6G) wireless systems. We propose a novel semantic communication framework by decomposing the metaverse into human/machine agent-specific semantic multiverses (SMs) An SM stored at each agent comprises a semantic encoder and a generator, leveraging recent advances in generative artificial intelligence (AI)
  arXiv  Detail & Related papers  (2022-12-13T21:21:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.