Attention-based Adversarial Robust Distillation in Radio Signal Classifications for Low-Power IoT Devices

• URL: http://arxiv.org/abs/2506.11892v1
• Date: Fri, 13 Jun 2025 15:39:01 GMT
• Title: Attention-based Adversarial Robust Distillation in Radio Signal Classifications for Low-Power IoT Devices
• Authors: Lu Zhang, Sangarapillai Lambotharan, Gan Zheng, Guisheng Liao, Basil AsSadhan, Fabio Roli,
• Abstract summary: We show that transformer-based radio signal classification is vulnerable to imperceptible and carefully crafted attacks called adversarial examples.<n>We propose a defense system against adversarial examples in transformer-based modulation classifications.<n>New method is aimed at transferring the adversarial attention map from the robustly trained large transformer to a compact transformer.
• Score: 28.874452850832213
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Due to great success of transformers in many applications such as natural language processing and computer vision, transformers have been successfully applied in automatic modulation classification. We have shown that transformer-based radio signal classification is vulnerable to imperceptible and carefully crafted attacks called adversarial examples. Therefore, we propose a defense system against adversarial examples in transformer-based modulation classifications. Considering the need for computationally efficient architecture particularly for Internet of Things (IoT)-based applications or operation of devices in environment where power supply is limited, we propose a compact transformer for modulation classification. The advantages of robust training such as adversarial training in transformers may not be attainable in compact transformers. By demonstrating this, we propose a novel compact transformer that can enhance robustness in the presence of adversarial attacks. The new method is aimed at transferring the adversarial attention map from the robustly trained large transformer to a compact transformer. The proposed method outperforms the state-of-the-art techniques for the considered white-box scenarios including fast gradient method and projected gradient descent attacks. We have provided reasoning of the underlying working mechanisms and investigated the transferability of the adversarial examples between different architectures. The proposed method has the potential to protect the transformer from the transferability of adversarial examples.

Related papers

• Vision Transformer with Adversarial Indicator Token against Adversarial Attacks in Radio Signal Classifications [33.246218531386326]
  We propose a novel vision transformer (ViT) architecture by introducing a new concept known as adversarial indicator (AdvI) token to detect adversarial attacks.<n>We show the proposed AdvI token acts as a crucial element within the ViT, influencing attention weights and thereby highlighting regions or features in the input data that are potentially suspicious or anomalous.
  arXiv  Detail & Related papers  (2025-06-13T15:21:54Z)
• Adversarial Robustness of In-Context Learning in Transformers for Linear Regression [23.737606860443705]
  This work investigates the vulnerability of in-context learning in transformers to textithijacking attacks focusing on the setting of linear regression tasks. We first prove that single-layer linear transformers, known to implement gradient descent in-context, are non-robust and can be manipulated to output arbitrary predictions. We then demonstrate that adversarial training enhances transformers' robustness against hijacking attacks, even when just applied during finetuning.
  arXiv  Detail & Related papers  (2024-11-07T21:25:58Z)
• The Efficacy of Transformer-based Adversarial Attacks in Security Domains [0.7156877824959499]
  We evaluate the robustness of transformers to adversarial samples for system defenders and their adversarial strength for system attackers. Our work emphasizes the importance of studying transformer architectures for attacking and defending models in security domains.
  arXiv  Detail & Related papers  (2023-10-17T21:45:23Z)
• XAI for Transformers: Better Explanations through Conservative Propagation [60.67748036747221]
  We show that the gradient in a Transformer reflects the function only locally, and thus fails to reliably identify the contribution of input features to the prediction. Our proposal can be seen as a proper extension of the well-established LRP method to Transformers.
  arXiv  Detail & Related papers  (2022-02-15T10:47:11Z)
• DBIA: Data-free Backdoor Injection Attack against Transformer Networks [6.969019759456717]
  We propose DBIA, a data-free backdoor attack against the CV-oriented transformer networks. Our approach can embed backdoors with a high success rate and a low impact on the performance of the victim transformers.
  arXiv  Detail & Related papers  (2021-11-22T08:13:51Z)
• Blending Anti-Aliasing into Vision Transformer [57.88274087198552]
  discontinuous patch-wise tokenization process implicitly introduces jagged artifacts into attention maps. Aliasing effect occurs when discrete patterns are used to produce high frequency or continuous information, resulting in the indistinguishable distortions. We propose a plug-and-play Aliasing-Reduction Module(ARM) to alleviate the aforementioned issue.
  arXiv  Detail & Related papers  (2021-10-28T14:30:02Z)
• The Nuts and Bolts of Adopting Transformer in GANs [124.30856952272913]
  We investigate the properties of Transformer in the generative adversarial network (GAN) framework for high-fidelity image synthesis. Our study leads to a new alternative design of Transformers in GAN, a convolutional neural network (CNN)-free generator termed as STrans-G.
  arXiv  Detail & Related papers  (2021-10-25T17:01:29Z)
• Scalable Transformers for Neural Machine Translation [86.4530299266897]
  Transformer has been widely adopted in Neural Machine Translation (NMT) because of its large capacity and parallel training of sequence generation. We propose a novel scalable Transformers, which naturally contains sub-Transformers of different scales and have shared parameters. A three-stage training scheme is proposed to tackle the difficulty of training the scalable Transformers.
  arXiv  Detail & Related papers  (2021-06-04T04:04:10Z)
• A Survey on Visual Transformer [126.56860258176324]
  Transformer is a type of deep neural network mainly based on the self-attention mechanism. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages.
  arXiv  Detail & Related papers  (2020-12-23T09:37:54Z)
• The Cascade Transformer: an Application for Efficient Answer Sentence Selection [116.09532365093659]
  We introduce the Cascade Transformer, a technique to adapt transformer-based models into a cascade of rankers. When compared to a state-of-the-art transformer model, our approach reduces computation by 37% with almost no impact on accuracy.
  arXiv  Detail & Related papers  (2020-05-05T23:32:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.