Zero-Sacrifice Persistent-Robustness Adversarial Defense for Pre-Trained Encoders

• URL: http://arxiv.org/abs/2602.11204v1
• Date: Tue, 10 Feb 2026 07:41:34 GMT
• Title: Zero-Sacrifice Persistent-Robustness Adversarial Defense for Pre-Trained Encoders
• Authors: Zhuxin Lei, Ziyuan Yang, Yi Zhang,
• Abstract summary: We introduce Zero-Sacrifice Persistent-Robustness Adversarial Defense (ZePAD)<n>ZePAD is inspired by the inherent sensitivity of neural networks to data characteristics.<n>It uses two adversarially fine-tuned encoders to strengthen adversarial resistance.
• Score: 5.00483763729881
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The widespread use of publicly available pre-trained encoders from self-supervised learning (SSL) has exposed a critical vulnerability: their susceptibility to downstream-agnostic adversarial examples (DAEs), which are crafted without knowledge of the downstream tasks but capable of misleading downstream models. While several defense methods have been explored recently, they rely primarily on task-specific adversarial fine-tuning, which inevitably limits generalizability and causes catastrophic forgetting and deteriorates benign performance. Different with previous works, we propose a more rigorous defense goal that requires only a single tuning for diverse downstream tasks to defend against DAEs and preserve benign performance. To achieve this defense goal, we introduce Zero-Sacrifice Persistent-Robustness Adversarial Defense (ZePAD), which is inspired by the inherent sensitivity of neural networks to data characteristics. Specifically, ZePAD is a dual-branch structure, which consists of a Multi-Pattern Adversarial Enhancement Branch (MPAE-Branch) that uses two adversarially fine-tuned encoders to strengthen adversarial resistance. The Benign Memory Preservation Branch (BMP-Branch) is trained on local data to ensure adversarial robustness does not compromise benign performance. Surprisingly, we find that ZePAD can directly detect DAEs by evaluating branch confidence, without introducing any adversarial exsample identification task during training. Notably, by enriching feature diversity, our method enables a single adversarial fine-tuning to defend against DAEs across downstream tasks, thereby achieving persistent robustness. Extensive experiments on 11 SSL methods and 6 datasets validate its effectiveness. In certain cases, it achieves a 29.20% improvement in benign performance and a 73.86% gain in adversarial robustness, highlighting its zero-sacrifice property.

Related papers

• Debiased Dual-Invariant Defense for Adversarially Robust Person Re-Identification [52.63017280231648]
  Person re-identification (ReID) is a fundamental task in many real-world applications such as pedestrian trajectory tracking.<n>Person ReID models are highly susceptible to adversarial attacks, where imperceptible perturbations to pedestrian images can cause entirely incorrect predictions.<n>We propose a dual-invariant defense framework composed of two main phases.
  arXiv  Detail & Related papers  (2025-11-13T03:56:40Z)
• Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference [45.723695657400576]
  We argue that enhancing the diversity and coverage of counterattacks is crucial to improving adversarial robustness in test-time defense.<n>We propose Directional Orthogonal Counterattack (DOC), which augments counterattack optimization by incorporating gradient directions and momentum-based updates.<n>We present a directional sensitivity score based on averaged cosine similarity to boost DOC by improving example discrimination and adaptively modulating the counterattack strength.
  arXiv  Detail & Related papers  (2025-11-12T07:40:16Z)
• Deep Positive-Negative Prototypes for Adversarially Robust Discriminative Prototypical Learning [0.24999074238880484]
  We propose a novel framework named Adversarially trained Deep Positive-Negative Prototypes (Adv-DPNP)<n>Adv-DPNP integrates discriminative prototype-based learning with adversarial training.<n>We show that Adv-DPNP achieves the highest average accuracy across severities and corruption types.
  arXiv  Detail & Related papers  (2025-04-03T15:42:58Z)
• Meta Invariance Defense Towards Generalizable Robustness to Unknown Adversarial Attacks [62.036798488144306]
  Current defense mainly focuses on the known attacks, but the adversarial robustness to the unknown attacks is seriously overlooked. We propose an attack-agnostic defense method named Meta Invariance Defense (MID) We show that MID simultaneously achieves robustness to the imperceptible adversarial perturbations in high-level image classification and attack-suppression in low-level robust image regeneration.
  arXiv  Detail & Related papers  (2024-04-04T10:10:38Z)
• Securely Fine-tuning Pre-trained Encoders Against Adversarial Examples [28.947545367473086]
  We propose a two-stage adversarial fine-tuning approach aimed at enhancing the robustness of downstream models. Our experiments, conducted across ten self-supervised training methods and six datasets, demonstrate that Gen-AF attains high testing accuracy and robust testing accuracy against state-of-the-art DAEs.
  arXiv  Detail & Related papers  (2024-03-16T04:23:46Z)
• Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Off [107.35833747750446]
  adversarial examples can be crafted by adding imperceptible perturbations to legitimate documents. This vulnerability raises significant concerns about their reliability and hinders the widespread deployment of NRMs. In this study, we establish theoretical guarantees regarding the effectiveness-robustness trade-off in NRMs.
  arXiv  Detail & Related papers  (2023-12-16T05:38:39Z)
• Improving Adversarial Robustness to Sensitivity and Invariance Attacks with Deep Metric Learning [80.21709045433096]
  A standard method in adversarial robustness assumes a framework to defend against samples crafted by minimally perturbing a sample. We use metric learning to frame adversarial regularization as an optimal transport problem. Our preliminary results indicate that regularizing over invariant perturbations in our framework improves both invariant and sensitivity defense.
  arXiv  Detail & Related papers  (2022-11-04T13:54:02Z)
• Exploring Robustness of Unsupervised Domain Adaptation in Semantic Segmentation [74.05906222376608]
  We propose adversarial self-supervision UDA (or ASSUDA) that maximizes the agreement between clean images and their adversarial examples by a contrastive loss in the output space. This paper is rooted in two observations: (i) the robustness of UDA methods in semantic segmentation remains unexplored, which pose a security concern in this field; and (ii) although commonly used self-supervision (e.g., rotation and jigsaw) benefits image tasks such as classification and recognition, they fail to provide the critical supervision signals that could learn discriminative representation for segmentation tasks.
  arXiv  Detail & Related papers  (2021-05-23T01:50:44Z)
• Rethinking Uncertainty in Deep Learning: Whether and How it Improves Robustness [20.912492996647888]
  adversarial training (AT) suffers from poor performance both on clean examples and under other types of attacks. Regularizers that encourage uncertain outputs, such as entropy (EntM) and label smoothing (LS) can maintain accuracy on clean examples and improve performance under weak attacks. In this paper, we revisit uncertainty promotion regularizers, including EntM and LS, in the field of adversarial learning.
  arXiv  Detail & Related papers  (2020-11-27T03:22:50Z)
• A Self-supervised Approach for Adversarial Robustness [105.88250594033053]
  Adversarial examples can cause catastrophic mistakes in Deep Neural Network (DNNs) based vision systems. This paper proposes a self-supervised adversarial training mechanism in the input space. It provides significant robustness against the textbfunseen adversarial attacks.
  arXiv  Detail & Related papers  (2020-06-08T20:42:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.