Robust Semantic Communications with Masked VQ-VAE Enabled Codebook
- URL: http://arxiv.org/abs/2206.04011v2
- Date: Wed, 19 Apr 2023 02:55:11 GMT
- Title: Robust Semantic Communications with Masked VQ-VAE Enabled Codebook
- Authors: Qiyu Hu, Guangyi Zhang, Zhijin Qin, Yunlong Cai, Guanding Yu, Geoffrey
Ye Li
- Abstract summary: We propose a framework for the robust end-to-end semantic communication systems to combat the semantic noise.
To combat the semantic noise, the adversarial training with weight is developed to incorporate the samples with semantic noise in the training dataset.
We develop a feature importance module (FIM) to suppress the noise-related and task-unrelated features.
- Score: 56.63571713657059
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Although semantic communications have exhibited satisfactory performance for
a large number of tasks, the impact of semantic noise and the robustness of the
systems have not been well investigated. Semantic noise refers to the
misleading between the intended semantic symbols and received ones, thus cause
the failure of tasks. In this paper, we first propose a framework for the
robust end-to-end semantic communication systems to combat the semantic noise.
In particular, we analyze sample-dependent and sample-independent semantic
noise. To combat the semantic noise, the adversarial training with weight
perturbation is developed to incorporate the samples with semantic noise in the
training dataset. Then, we propose to mask a portion of the input, where the
semantic noise appears frequently, and design the masked vector
quantized-variational autoencoder (VQ-VAE) with the noise-related masking
strategy. We use a discrete codebook shared by the transmitter and the receiver
for encoded feature representation. To further improve the system robustness,
we develop a feature importance module (FIM) to suppress the noise-related and
task-unrelated features. Thus, the transmitter simply needs to transmit the
indices of these important task-related features in the codebook. Simulation
results show that the proposed method can be applied in many downstream tasks
and significantly improve the robustness against semantic noise with remarkable
reduction on the transmission overhead.
Related papers
- One-step Noisy Label Mitigation [86.57572253460125]
Mitigating the detrimental effects of noisy labels on the training process has become increasingly critical.
We propose One-step Anti-Noise (OSA), a model-agnostic noisy label mitigation paradigm.
We empirically demonstrate the superiority of OSA, highlighting its enhanced training robustness, improved task transferability, ease of deployment, and reduced computational costs.
arXiv Detail & Related papers (2024-10-02T18:42:56Z) - Effective Noise-aware Data Simulation for Domain-adaptive Speech Enhancement Leveraging Dynamic Stochastic Perturbation [25.410770364140856]
Cross-domain speech enhancement (SE) is often faced with severe challenges due to the scarcity of noise and background information in an unseen target domain.
This study puts forward a novel data simulation method to address this issue, leveraging noise-extractive techniques and generative adversarial networks (GANs)
We introduce the notion of dynamic perturbation, which can inject controlled perturbations into the noise embeddings during inference.
arXiv Detail & Related papers (2024-09-03T02:29:01Z) - Learning to Correct Noisy Labels for Fine-Grained Entity Typing via
Co-Prediction Prompt Tuning [9.885278527023532]
We introduce Co-Prediction Prompt Tuning for noise correction in FET.
We integrate prediction results to recall labeled labels and utilize a differentiated margin to identify inaccurate labels.
Experimental results on three widely-used FET datasets demonstrate that our noise correction approach significantly enhances the quality of training samples.
arXiv Detail & Related papers (2023-10-23T06:04:07Z) - DINF: Dynamic Instance Noise Filter for Occluded Pedestrian Detection [0.0]
RCNN-based pedestrian detectors use rectangle regions to extract instance features.
The number of severely overlapping objects and the number of slightly overlapping objects are unbalanced.
An iterable dynamic instance noise filter (DINF) is proposed for the RCNN-based pedestrian detectors to improve the signal-noise ratio of the instance feature.
arXiv Detail & Related papers (2023-01-13T14:12:36Z) - NLIP: Noise-robust Language-Image Pre-training [95.13287735264937]
We propose a principled Noise-robust Language-Image Pre-training framework (NLIP) to stabilize pre-training via two schemes: noise-harmonization and noise-completion.
Our NLIP can alleviate the common noise effects during image-text pre-training in a more efficient way.
arXiv Detail & Related papers (2022-12-14T08:19:30Z) - Inference and Denoise: Causal Inference-based Neural Speech Enhancement [83.4641575757706]
This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention.
The proposed causal inference-based speech enhancement (CISE) separates clean and noisy frames in an intervened noisy speech using a noise detector and assigns both sets of frames to two mask-based enhancement modules (EMs) to perform noise-conditional SE.
arXiv Detail & Related papers (2022-11-02T15:03:50Z) - Deep Semantic Statistics Matching (D2SM) Denoising Network [70.01091467628068]
We introduce the Deep Semantic Statistics Matching (D2SM) Denoising Network.
It exploits semantic features of pretrained classification networks, then it implicitly matches the probabilistic distribution of clear images at the semantic feature space.
By learning to preserve the semantic distribution of denoised images, we empirically find our method significantly improves the denoising capabilities of networks.
arXiv Detail & Related papers (2022-07-19T14:35:42Z) - Robust Semantic Communications Against Semantic Noise [34.80426719511182]
We first propose a framework for the robust end-to-end semantic communication systems to combat semantic noise.
We analyze the causes of semantic noise and propose a practical method to generate it.
Our proposed method significantly improves the robustness of semantic communication systems against semantic noise with significant reduction on the transmission overhead.
arXiv Detail & Related papers (2022-02-07T16:37:45Z) - Improving Noise Robustness of Contrastive Speech Representation Learning
with Speech Reconstruction [109.44933866397123]
Noise robustness is essential for deploying automatic speech recognition systems in real-world environments.
We employ a noise-robust representation learned by a refined self-supervised framework for noisy speech recognition.
We achieve comparable performance to the best supervised approach reported with only 16% of labeled data.
arXiv Detail & Related papers (2021-10-28T20:39:02Z) - Timbre Transfer with Variational Auto Encoding and Cycle-Consistent
Adversarial Networks [0.6445605125467573]
This research project investigates the application of deep learning to timbre transfer, where the timbre of a source audio can be converted to the timbre of a target audio with minimal loss in quality.
The adopted approach combines Variational Autoencoders with Generative Adversarial Networks to construct meaningful representations of the source audio and produce realistic generations of the target audio.
arXiv Detail & Related papers (2021-09-05T15:06:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.