Generative Semantic Communication for Joint Image Transmission and Segmentation

• URL: http://arxiv.org/abs/2411.18005v1
• Date: Wed, 27 Nov 2024 02:51:26 GMT
• Title: Generative Semantic Communication for Joint Image Transmission and Segmentation
• Authors: Weiwen Yuan, Jinke Ren, Chongjie Wang, Ruichen Zhang, Jun Wei, Dong In Kim, Shuguang Cui,
• Abstract summary: We propose a novel generative semantic communication system that supports both image reconstruction and segmentation tasks. Our approach builds upon semantic knowledge bases (KBs) at both the transmitter and receiver. Experimental results demonstrate that our multi-task generative semantic communication system outperforms previous single-task communication systems.
• Score: 39.39101766098789
• License:
• Abstract: Semantic communication has emerged as a promising technology for enhancing communication efficiency. However, most existing research emphasizes single-task reconstruction, neglecting model adaptability and generalization across multi-task systems. In this paper, we propose a novel generative semantic communication system that supports both image reconstruction and segmentation tasks. Our approach builds upon semantic knowledge bases (KBs) at both the transmitter and receiver, with each semantic KB comprising a source KB and a task KB. The source KB at the transmitter leverages a hierarchical Swin-Transformer, a generative AI scheme, to extract multi-level features from the input image. Concurrently, the counterpart source KB at the receiver utilizes hierarchical residual blocks to generate task-specific knowledge. Furthermore, the two task KBs adopt a semantic similarity model to map different task requirements into pre-defined task instructions, thereby facilitating the feature selection of the source KBs. Additionally, we develop a unified residual block-based joint source and channel (JSCC) encoder and two task-specific JSCC decoders to achieve the two image tasks. In particular, a generative diffusion model is adopted to construct the JSCC decoder for the image reconstruction task. Experimental results demonstrate that our multi-task generative semantic communication system outperforms previous single-task communication systems in terms of peak signal-to-noise ratio and segmentation accuracy.

Related papers

• Take What You Need: Flexible Multi-Task Semantic Communications with Channel Adaptation [51.53221300103261]
  This article introduces a novel channel-adaptive and multi-task-aware semantic communication framework based on a masked auto-encoder architecture. A channel-aware extractor is employed to dynamically select relevant information in response to real-time channel conditions. Experimental results demonstrate the superior performance of our framework compared to conventional methods in tasks such as image reconstruction and object detection.
  arXiv  Detail & Related papers  (2025-02-12T09:01:25Z)
• Multi-Task Semantic Communication With Graph Attention-Based Feature Correlation Extraction [69.24689059980035]
  This paper presents a new graph attention inter-block (GAI) module to the encoder/transmitter of a multi-task semantic communication system. We interpret the outputs of the intermediate feature extraction blocks of the encoder as the nodes of a graph to capture the correlations of the intermediate features. Experiments demonstrate that the proposed model surpasses the most competitive and publicly available models by 11.4% on the CityScapes 2Task dataset.
  arXiv  Detail & Related papers  (2025-01-02T04:38:01Z)
• Vision Transformer-based Semantic Communications With Importance-Aware Quantization [13.328970689723096]
  This paper presents a vision transformer (ViT)-based semantic communication system with importance-aware quantization (IAQ) for wireless image transmission. We show that our IAQ framework outperforms conventional image compression methods in both error-free and realistic communication scenarios.
  arXiv  Detail & Related papers  (2024-12-08T19:24:47Z)
• Generative Semantic Communication for Text-to-Speech Synthesis [39.8799066368712]
  This paper develops a novel generative semantic communication framework for text-to-speech synthesis. We employ a transformer encoder and a diffusion model to achieve efficient semantic coding without introducing significant communication overhead.
  arXiv  Detail & Related papers  (2024-10-04T14:18:31Z)
• Trustworthy Image Semantic Communication with GenAI: Explainablity, Controllability, and Efficiency [59.15544887307901]
  Image semantic communication (ISC) has garnered significant attention for its potential to achieve high efficiency in visual content transmission. Existing ISC systems based on joint source-channel coding face challenges in interpretability, operability, and compatibility. We propose a novel trustworthy ISC framework that employs Generative Artificial Intelligence (GenAI) for multiple downstream inference tasks.
  arXiv  Detail & Related papers  (2024-08-07T14:32:36Z)
• Agent-driven Generative Semantic Communication with Cross-Modality and Prediction [57.335922373309074]
  We propose a novel agent-driven generative semantic communication framework based on reinforcement learning. In this work, we develop an agent-assisted semantic encoder with cross-modality capability, which can track the semantic changes, channel condition, to perform adaptive semantic extraction and sampling. The effectiveness of the designed models has been verified using the UA-DETRAC dataset, demonstrating the performance gains of the overall A-GSC framework.
  arXiv  Detail & Related papers  (2024-04-10T13:24:27Z)
• A Multi-Task Oriented Semantic Communication Framework for Autonomous Vehicles [5.779316179788962]
  This work presents a multi-task-oriented semantic communication framework for connected and autonomous vehicles. We propose a convolutional autoencoder (CAE) that performs the semantic encoding of the road traffic signs. These encoded images are then transmitted from one CAV to another CAV through satellite in challenging weather conditions.
  arXiv  Detail & Related papers  (2024-03-06T12:04:24Z)
• Multi-Receiver Task-Oriented Communications via Multi-Task Deep Learning [49.83882366499547]
  This paper studies task-oriented, otherwise known as goal-oriented, communications in a setting where a transmitter communicates with multiple receivers. A multi-task deep learning approach is presented for joint optimization of completing multiple tasks and communicating with multiple receivers.
  arXiv  Detail & Related papers  (2023-08-14T01:34:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.