Synchronous Multi-modal Semantic Communication System with Packet-level Coding

• URL: http://arxiv.org/abs/2408.04535v2
• Date: Sun, 11 Aug 2024 02:37:42 GMT
• Title: Synchronous Multi-modal Semantic Communication System with Packet-level Coding
• Authors: Yun Tian, Jingkai Ying, Zhijin Qin, Ye Jin, Xiaoming Tao,
• Abstract summary: We propose a Synchronous Multimodal Semantic Communication System (SyncSC) with Packet-Level Coding. To achieve semantic and time synchronization, 3D Morphable Mode (3DMM) coefficients and text are transmitted as semantics. To protect semantic packets under the erasure channel, we propose a packet-Level Forward Error Correction (FEC) method, called PacSC, that maintains a certain visual quality performance even at high packet loss rates.
• Score: 20.397350999784276
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although the semantic communication with joint semantic-channel coding design has shown promising performance in transmitting data of different modalities over physical layer channels, the synchronization and packet-level forward error correction of multimodal semantics have not been well studied. Due to the independent design of semantic encoders, synchronizing multimodal features in both the semantic and time domains is a challenging problem. In this paper, we take the facial video and speech transmission as an example and propose a Synchronous Multimodal Semantic Communication System (SyncSC) with Packet-Level Coding. To achieve semantic and time synchronization, 3D Morphable Mode (3DMM) coefficients and text are transmitted as semantics, and we propose a semantic codec that achieves similar quality of reconstruction and synchronization with lower bandwidth, compared to traditional methods. To protect semantic packets under the erasure channel, we propose a packet-Level Forward Error Correction (FEC) method, called PacSC, that maintains a certain visual quality performance even at high packet loss rates. Particularly, for text packets, a text packet loss concealment module, called TextPC, based on Bidirectional Encoder Representations from Transformers (BERT) is proposed, which significantly improves the performance of traditional FEC methods. The simulation results show that our proposed SyncSC reduce transmission overhead and achieve high-quality synchronous transmission of video and speech over the packet loss network.

Related papers

• Generative Video Semantic Communication via Multimodal Semantic Fusion with Large Model [55.71885688565501]
  We propose a scalable generative video semantic communication framework that extracts and transmits semantic information to achieve high-quality video reconstruction. Specifically, at the transmitter, description and other condition signals are extracted from the source video, functioning as text and structural semantics, respectively. At the receiver, the diffusion-based GenAI large models are utilized to fuse the semantics of the multiple modalities for reconstructing the video.
  arXiv  Detail & Related papers  (2025-02-19T15:59:07Z)
• Take What You Need: Flexible Multi-Task Semantic Communications with Channel Adaptation [51.53221300103261]
  This article introduces a novel channel-adaptive and multi-task-aware semantic communication framework based on a masked auto-encoder architecture. A channel-aware extractor is employed to dynamically select relevant information in response to real-time channel conditions. Experimental results demonstrate the superior performance of our framework compared to conventional methods in tasks such as image reconstruction and object detection.
  arXiv  Detail & Related papers  (2025-02-12T09:01:25Z)
• Cross-Layer Encrypted Semantic Communication Framework for Panoramic Video Transmission [11.438045765196332]
  We propose a cross-layer encrypted semantic communication (CLESC) framework for panoramic video transmission. We propose an adaptive cross-layer transmission mechanism that dynamically adjusts CRC, channel coding, and retransmission schemes based on the importance of semantic information. Compared to traditional cross-layer transmission schemes, the CLESC framework can reduce bandwidth consumption by 85%.
  arXiv  Detail & Related papers  (2024-11-19T07:18:38Z)
• Visual Language Model based Cross-modal Semantic Communication Systems [42.321208020228894]
  We propose a novel Vision-Language Model-based Cross-modal Semantic Communication system. The VLM-CSC comprises three novel components. The experimental simulations validate the effectiveness, adaptability, and robustness of the CSC system.
  arXiv  Detail & Related papers  (2024-05-06T08:59:16Z)
• Latency-Aware Generative Semantic Communications with Pre-Trained Diffusion Models [43.27015039765803]
  We develop a latency-aware semantic communications framework with pre-trained generative models. We demonstrate ultra-low-rate, low-latency, and channel-adaptive semantic communications.
  arXiv  Detail & Related papers  (2024-03-25T23:04:09Z)
• Generative AI-aided Joint Training-free Secure Semantic Communications via Multi-modal Prompts [89.04751776308656]
  This paper proposes a GAI-aided SemCom system with multi-model prompts for accurate content decoding. In response to security concerns, we introduce the application of covert communications aided by a friendly jammer.
  arXiv  Detail & Related papers  (2023-09-05T23:24:56Z)
• Communication-Efficient Framework for Distributed Image Semantic Wireless Transmission [68.69108124451263]
  Federated learning-based semantic communication (FLSC) framework for multi-task distributed image transmission with IoT devices. Each link is composed of a hierarchical vision transformer (HVT)-based extractor and a task-adaptive translator. Channel state information-based multiple-input multiple-output transmission module designed to combat channel fading and noise.
  arXiv  Detail & Related papers  (2023-08-07T16:32:14Z)
• Enabling the Wireless Metaverse via Semantic Multiverse Communication [82.47169682083806]
  Metaverse over wireless networks is an emerging use case of the sixth generation (6G) wireless systems. We propose a novel semantic communication framework by decomposing the metaverse into human/machine agent-specific semantic multiverses (SMs) An SM stored at each agent comprises a semantic encoder and a generator, leveraging recent advances in generative artificial intelligence (AI)
  arXiv  Detail & Related papers  (2022-12-13T21:21:07Z)
• Semantic Image Synthesis via Diffusion Models [159.4285444680301]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks. Recent work on semantic image synthesis mainly follows the emphde facto Generative Adversarial Nets (GANs)
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• Wireless Deep Video Semantic Transmission [14.071114007641313]
  We propose a new class of high-efficiency deep joint source-channel coding methods to achieve end-to-end video transmission over wireless channels. Our framework is collected under the name deep video semantic transmission (DVST)
  arXiv  Detail & Related papers  (2022-05-26T03:26:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.