Semantic Feature Decomposition based Semantic Communication System of Images with Large-scale Visual Generation Models

• URL: http://arxiv.org/abs/2410.20126v1
• Date: Sat, 26 Oct 2024 08:53:05 GMT
• Title: Semantic Feature Decomposition based Semantic Communication System of Images with Large-scale Visual Generation Models
• Authors: Senran Fan, Zhicheng Bao, Chen Dong, Haotai Liang, Xiaodong Xu, Ping Zhang,
• Abstract summary: A Texture-Color based Semantic Communication system of Images TCSCI is proposed. It decomposing the images into their natural language description (text), texture and color semantic features at the transmitter. It can achieve extremely compressed, highly noise-resistant, and visually similar image semantic communication, while ensuring the interpretability and editability of the transmission process.
• Score: 5.867765921443141
• License:
• Abstract: The end-to-end image communication system has been widely studied in the academic community. The escalating demands on image communication systems in terms of data volume, environmental complexity, and task precision require enhanced communication efficiency, anti-noise ability and semantic fidelity. Therefore, we proposed a novel paradigm based on Semantic Feature Decomposition (SeFD) for the integration of semantic communication and large-scale visual generation models to achieve high-performance, highly interpretable and controllable image communication. According to this paradigm, a Texture-Color based Semantic Communication system of Images TCSCI is proposed. TCSCI decomposing the images into their natural language description (text), texture and color semantic features at the transmitter. During the transmission, features are transmitted over the wireless channel, and at the receiver, a large-scale visual generation model is utilized to restore the image through received features. TCSCI can achieve extremely compressed, highly noise-resistant, and visually similar image semantic communication, while ensuring the interpretability and editability of the transmission process. The experiments demonstrate that the TCSCI outperforms traditional image communication systems and existing semantic communication systems under extreme compression with good anti-noise performance and interpretability.

Related papers

• Large Generative Model-assisted Talking-face Semantic Communication System [55.42631520122753]
  This study introduces a Large Generative Model-assisted Talking-face Semantic Communication (LGM-TSC) system. Generative Semantic Extractor (GSE) at the transmitter converts semantically sparse talking-face videos into texts with high information density. Private Knowledge Base (KB) based on the Large Language Model (LLM) for semantic disambiguation and correction. Generative Semantic Reconstructor (GSR) that utilizes BERT-VITS2 and SadTalker models to transform text back into a high-QoE talking-face video.
  arXiv  Detail & Related papers  (2024-11-06T12:45:46Z)
• Trustworthy Image Semantic Communication with GenAI: Explainablity, Controllability, and Efficiency [59.15544887307901]
  Image semantic communication (ISC) has garnered significant attention for its potential to achieve high efficiency in visual content transmission. Existing ISC systems based on joint source-channel coding face challenges in interpretability, operability, and compatibility. We propose a novel trustworthy ISC framework that employs Generative Artificial Intelligence (GenAI) for multiple downstream inference tasks.
  arXiv  Detail & Related papers  (2024-08-07T14:32:36Z)
• Semantic Similarity Score for Measuring Visual Similarity at Semantic Level [5.867765921443141]
  We propose a semantic evaluation metric -- SeSS (Semantic Similarity Score) based on Scene Graph Generation and graph matching. The metric can measure the semantic-level differences in semantic-level information of images and can be used for evaluation in visual semantic communication systems.
  arXiv  Detail & Related papers  (2024-06-06T08:51:26Z)
• Deep Joint Semantic Coding and Beamforming for Near-Space Airship-Borne Massive MIMO Network [70.63240823677182]
  Near-space airship-borne communication network urgently needs reliable and efficient Airship-to-X link. This paper proposes to integrate semantic communication with massive multiple-input multiple-output (MIMO) technology.
  arXiv  Detail & Related papers  (2024-05-30T09:46:59Z)
• Generative AI-aided Joint Training-free Secure Semantic Communications via Multi-modal Prompts [89.04751776308656]
  This paper proposes a GAI-aided SemCom system with multi-model prompts for accurate content decoding. In response to security concerns, we introduce the application of covert communications aided by a friendly jammer.
  arXiv  Detail & Related papers  (2023-09-05T23:24:56Z)
• Communication-Efficient Framework for Distributed Image Semantic Wireless Transmission [68.69108124451263]
  Federated learning-based semantic communication (FLSC) framework for multi-task distributed image transmission with IoT devices. Each link is composed of a hierarchical vision transformer (HVT)-based extractor and a task-adaptive translator. Channel state information-based multiple-input multiple-output transmission module designed to combat channel fading and noise.
  arXiv  Detail & Related papers  (2023-08-07T16:32:14Z)
• Causal Semantic Communication for Digital Twins: A Generalizable Imitation Learning Approach [74.25870052841226]
  A digital twin (DT) leverages a virtual representation of the physical world, along with communication (e.g., 6G), computing, and artificial intelligence (AI) technologies to enable many connected intelligence services. Wireless systems can exploit the paradigm of semantic communication (SC) for facilitating informed decision-making under strict communication constraints. A novel framework called causal semantic communication (CSC) is proposed for DT-based wireless systems.
  arXiv  Detail & Related papers  (2023-04-25T00:15:00Z)
• Wireless End-to-End Image Transmission System using Semantic Communications [4.2421412410466575]
  The research shows that the resource gain in the form of bandwidth saving is immense when transmitting the semantic segmentation map through the physical channel. The research studies the effect of physical channel distortions and quantization noise on semantic communication-based multimedia content transmission.
  arXiv  Detail & Related papers  (2023-02-27T12:33:53Z)
• Vector Quantized Semantic Communication System [22.579525825992416]
  We develop a deep learning-enabled vector quantized (VQ) semantic communication system for image transmission, named VQ-DeepSC. Specifically, we propose a CNN-based transceiver to extract multi-scale semantic features of images and introduce multi-scale semantic embedding spaces. We employ adversarial training to improve the quality of received images by introducing a PatchGAN discriminator.
  arXiv  Detail & Related papers  (2022-09-23T10:58:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.