Knowledge Distillation Based Semantic Communications For Multiple Users
- URL: http://arxiv.org/abs/2311.13789v1
- Date: Thu, 23 Nov 2023 03:28:14 GMT
- Title: Knowledge Distillation Based Semantic Communications For Multiple Users
- Authors: Chenguang Liu, Yuxin Zhou, Yunfei Chen and Shuang-Hua Yang
- Abstract summary: We consider the semantic communication (SemCom) system with multiple users, where there is a limited number of training samples and unexpected interference.
We propose a knowledge distillation (KD) based system where Transformer based encoder-decoder is implemented as the semantic encoder-decoder and fully connected neural networks are implemented as the channel encoder-decoder.
Numerical results demonstrate that KD significantly improves the robustness and the generalization ability when applied to unexpected interference, and it reduces the performance loss when compressing the model size.
- Score: 10.770552656390038
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Deep learning (DL) has shown great potential in revolutionizing the
traditional communications system. Many applications in communications have
adopted DL techniques due to their powerful representation ability. However,
the learning-based methods can be dependent on the training dataset and perform
worse on unseen interference due to limited model generalizability and
complexity. In this paper, we consider the semantic communication (SemCom)
system with multiple users, where there is a limited number of training samples
and unexpected interference. To improve the model generalization ability and
reduce the model size, we propose a knowledge distillation (KD) based system
where Transformer based encoder-decoder is implemented as the semantic
encoder-decoder and fully connected neural networks are implemented as the
channel encoder-decoder. Specifically, four types of knowledge transfer and
model compression are analyzed. Important system and model parameters are
considered, including the level of noise and interference, the number of
interfering users and the size of the encoder and decoder. Numerical results
demonstrate that KD significantly improves the robustness and the
generalization ability when applied to unexpected interference, and it reduces
the performance loss when compressing the model size.
Related papers
- Dynamic Encoding and Decoding of Information for Split Learning in
Mobile-Edge Computing: Leveraging Information Bottleneck Theory [1.1151919978983582]
Split learning is a privacy-preserving distributed learning paradigm in which an ML model is split into two parts (i.e., an encoder and a decoder)
In mobile-edge computing, network functions can be trained via split learning where an encoder resides in a user equipment (UE) and a decoder resides in the edge network.
We present a new framework and training mechanism to enable a dynamic balancing of the transmission resource consumption with the informativeness of the shared latent representations.
arXiv Detail & Related papers (2023-09-06T07:04:37Z) - Causal Semantic Communication for Digital Twins: A Generalizable
Imitation Learning Approach [74.25870052841226]
A digital twin (DT) leverages a virtual representation of the physical world, along with communication (e.g., 6G), computing, and artificial intelligence (AI) technologies to enable many connected intelligence services.
Wireless systems can exploit the paradigm of semantic communication (SC) for facilitating informed decision-making under strict communication constraints.
A novel framework called causal semantic communication (CSC) is proposed for DT-based wireless systems.
arXiv Detail & Related papers (2023-04-25T00:15:00Z) - Enhancing Multiple Reliability Measures via Nuisance-extended
Information Bottleneck [77.37409441129995]
In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition.
We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training.
We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
arXiv Detail & Related papers (2023-03-24T16:03:21Z) - Model-based Deep Learning Receiver Design for Rate-Splitting Multiple
Access [65.21117658030235]
This work proposes a novel design for a practical RSMA receiver based on model-based deep learning (MBDL) methods.
The MBDL receiver is evaluated in terms of uncoded Symbol Error Rate (SER), throughput performance through Link-Level Simulations (LLS) and average training overhead.
Results reveal that the MBDL outperforms by a significant margin the SIC receiver with imperfect CSIR.
arXiv Detail & Related papers (2022-05-02T12:23:55Z) - Adaptive Discrete Communication Bottlenecks with Dynamic Vector
Quantization [76.68866368409216]
We propose learning to dynamically select discretization tightness conditioned on inputs.
We show that dynamically varying tightness in communication bottlenecks can improve model performance on visual reasoning and reinforcement learning tasks.
arXiv Detail & Related papers (2022-02-02T23:54:26Z) - Discrete-Valued Neural Communication [85.3675647398994]
We show that restricting the transmitted information among components to discrete representations is a beneficial bottleneck.
Even though individuals have different understandings of what a "cat" is based on their specific experiences, the shared discrete token makes it possible for communication among individuals to be unimpeded by individual differences in internal representation.
We extend the quantization mechanism from the Vector-Quantized Variational Autoencoder to multi-headed discretization with shared codebooks and use it for discrete-valued neural communication.
arXiv Detail & Related papers (2021-07-06T03:09:25Z) - Capacity-Approaching Autoencoders for Communications [4.86067125387358]
The current approach to train an autoencoder relies on the use of the cross-entropy loss function.
We propose a methodology that computes an estimate of the channel capacity and constructs an optimal coded signal approaching it.
arXiv Detail & Related papers (2020-09-11T08:19:06Z) - Neural Communication Systems with Bandwidth-limited Channel [9.332315420944836]
Reliably transmitting messages despite information loss is a core problem of information theory.
In this study we consider learning coding with the bandwidth-limited channel (BWLC)
arXiv Detail & Related papers (2020-03-30T11:58:30Z) - Data-Driven Symbol Detection via Model-Based Machine Learning [117.58188185409904]
We review a data-driven framework to symbol detection design which combines machine learning (ML) and model-based algorithms.
In this hybrid approach, well-known channel-model-based algorithms are augmented with ML-based algorithms to remove their channel-model-dependence.
Our results demonstrate that these techniques can yield near-optimal performance of model-based algorithms without knowing the exact channel input-output statistical relationship.
arXiv Detail & Related papers (2020-02-14T06:58:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.