Rate-Adaptive Coding Mechanism for Semantic Communications With
Multi-Modal Data
- URL: http://arxiv.org/abs/2305.10773v1
- Date: Thu, 18 May 2023 07:31:37 GMT
- Title: Rate-Adaptive Coding Mechanism for Semantic Communications With
Multi-Modal Data
- Authors: Yangshuo He, Guanding Yu, Yunlong Cai
- Abstract summary: We propose a distributed multi-modal semantic communication framework incorporating the conventional channel encoder/decoder.
We establish a general rate-adaptive coding mechanism for various types of multi-modal semantic tasks.
Numerical results show that the proposed mechanism fares better than both conventional communication and existing semantic communication systems.
- Score: 23.597759255020296
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recently, the ever-increasing demand for bandwidth in multi-modal
communication systems requires a paradigm shift. Powered by deep learning,
semantic communications are applied to multi-modal scenarios to boost
communication efficiency and save communication resources. However, the
existing end-to-end neural network (NN) based framework without the channel
encoder/decoder is incompatible with modern digital communication systems.
Moreover, most end-to-end designs are task-specific and require re-design and
re-training for new tasks, which limits their applications. In this paper, we
propose a distributed multi-modal semantic communication framework
incorporating the conventional channel encoder/decoder. We adopt NN-based
semantic encoder and decoder to extract correlated semantic information
contained in different modalities, including speech, text, and image. Based on
the proposed framework, we further establish a general rate-adaptive coding
mechanism for various types of multi-modal semantic tasks. In particular, we
utilize unequal error protection based on semantic importance, which is derived
by evaluating the distortion bound of each modality. We further formulate and
solve an optimization problem that aims at minimizing inference delay while
maintaining inference accuracy for semantic tasks. Numerical results show that
the proposed mechanism fares better than both conventional communication and
existing semantic communication systems in terms of task performance, inference
delay, and deployment complexity.
Related papers
- Adaptive Semantic Token Selection for AI-native Goal-oriented Communications [11.92172357956248]
We propose a novel design for AI-native goal-oriented communications.
We exploit transformer neural networks under dynamic inference constraints on bandwidth and computation.
We show that our model improves over state-of-the-art token selection mechanisms.
arXiv Detail & Related papers (2024-04-25T13:49:50Z) - Agent-driven Generative Semantic Communication with Cross-Modality and Prediction [57.335922373309074]
We propose a novel agent-driven generative semantic communication framework based on reinforcement learning.
In this work, we develop an agent-assisted semantic encoder with cross-modality capability, which can track the semantic changes, channel condition, to perform adaptive semantic extraction and sampling.
The effectiveness of the designed models has been verified using the UA-DETRAC dataset, demonstrating the performance gains of the overall A-GSC framework.
arXiv Detail & Related papers (2024-04-10T13:24:27Z) - Adaptive Resource Allocation for Semantic Communication Networks [34.189531352110386]
This paper investigates the quality of service for semantic communication networks, including the semantic quantization efficiency (SQE) and transmission latency.
A problem maximizing the overall effective SC-QoS is formulated by jointly the transmit beamforming the base station, the bits semantic representation the subchannel assignment, and the semantic resource allocation.
Our design can effectively combat semantic noise and achieve superior performance in wireless communications compared to several benchmark schemes.
arXiv Detail & Related papers (2023-12-02T09:12:12Z) - Reasoning with the Theory of Mind for Pragmatic Semantic Communication [62.87895431431273]
A pragmatic semantic communication framework is proposed in this paper.
It enables effective goal-oriented information sharing between two-intelligent agents.
Numerical evaluations demonstrate the framework's ability to achieve efficient communication with a reduced amount of bits.
arXiv Detail & Related papers (2023-11-30T03:36:19Z) - Generative AI-aided Joint Training-free Secure Semantic Communications
via Multi-modal Prompts [89.04751776308656]
This paper proposes a GAI-aided SemCom system with multi-model prompts for accurate content decoding.
In response to security concerns, we introduce the application of covert communications aided by a friendly jammer.
arXiv Detail & Related papers (2023-09-05T23:24:56Z) - Communication-Efficient Framework for Distributed Image Semantic
Wireless Transmission [68.69108124451263]
Federated learning-based semantic communication (FLSC) framework for multi-task distributed image transmission with IoT devices.
Each link is composed of a hierarchical vision transformer (HVT)-based extractor and a task-adaptive translator.
Channel state information-based multiple-input multiple-output transmission module designed to combat channel fading and noise.
arXiv Detail & Related papers (2023-08-07T16:32:14Z) - Cognitive Semantic Communication Systems Driven by Knowledge Graph:
Principle, Implementation, and Performance Evaluation [74.38561925376996]
Two cognitive semantic communication frameworks are proposed for the single-user and multiple-user communication scenarios.
An effective semantic correction algorithm is proposed by mining the inference rule from the knowledge graph.
For the multi-user cognitive semantic communication system, a message recovery algorithm is proposed to distinguish messages of different users.
arXiv Detail & Related papers (2023-03-15T12:01:43Z) - Non-Orthogonal Multiple Access Enhanced Multi-User Semantic
Communication [7.231199109999097]
Non-orthogonal multiple access (NOMA)-based multi-user semantic communication system named NOMASC is proposed in this paper.
System can support semantic tranmission of multiple users with diverse modalities of source information.
arXiv Detail & Related papers (2023-03-12T07:32:42Z) - Emergent Quantized Communication [34.31732248872158]
We propose an alternative approach to achieve discrete communication -- quantization of communicated messages.
Message quantization allows us to train the model end-to-end, achieving superior performance in multiple setups.
arXiv Detail & Related papers (2022-11-04T12:39:45Z) - Deep Multi-Task Learning for Cooperative NOMA: System Design and
Principles [52.79089414630366]
We develop a novel deep cooperative NOMA scheme, drawing upon the recent advances in deep learning (DL)
We develop a novel hybrid-cascaded deep neural network (DNN) architecture such that the entire system can be optimized in a holistic manner.
arXiv Detail & Related papers (2020-07-27T12:38:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.