Information Theoretic Adversarial Training of Large Language Models

• URL: http://arxiv.org/abs/2605.05415v1
• Date: Wed, 06 May 2026 20:20:09 GMT
• Title: Information Theoretic Adversarial Training of Large Language Models
• Abstract summary: WARDEN is a distributionally robust adversarial training framework for large language models.<n>It dynamically reweights adversarial examples through an f -divergence ambiguity set around the empirical training distribution.<n>It substantially reduces attack success rates with computational and utility costs comparable to CAT-, CAPO-, and MixAT-based baselines.
• Score: 26.056920129810294
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models (LLMs) remain vulnerable to adversarial prompting despite advances in alignment and safety, often exhibiting harmful behaviors under novel attack strategies. While adversarial training can improve robustness, existing approaches are computationally expensive and difficult to scale. Recent continuous adversarial training methods, such as Continuous adversarial training (CAT) and Continuous Adversarial Preference Optimization (CAPO), address this challenge by leveraging gradient-based perturbations in the embedding space, enabling more efficient and expressive attacks. Building on this paradigm, we propose WARDEN, a distributionally robust adversarial training framework for LLMs that dynamically reweights adversarial examples through an f -divergence ambiguity set around the empirical training distribution. Our method optimizes the worst-case adversarial loss within a divergence ball around the empirical data distribution, automatically emphasizing harder adversarial examples. Using the convex dual formulation, the objective reduces to a log-sum-exp form under the KL divergence, with a dynamical parameter controlling the strength of reweighting. This study leads to a new class of information-theoretic objectives that significantly reduce attack success rates while maintaining model utility. Across multiple LLMs and attack settings, WARDEN substantially reduces attack success rates with computational and utility costs comparable to CAT-, CAPO-, and MixAT-based baselines, making it a practical approach for scalable robust alignment.

Related papers

• Low Rank Adaptation for Adversarial Perturbation [50.24377354110586]
  Low-Rank Adaptation (LoRA) has significantly improved the training efficiency of Large Language Models (LLMs)<n>Since the generation of adversarial examples is an optimization process analogous to model training, this naturally raises the question: Do adversarial perturbations exhibit a similar low-rank structure?<n>In this paper, we provide both theoretical analysis and extensive empirical investigation across various attack methods, model architectures, and datasets to show that adversarial perturbations indeed possess a low-rank structure.
  arXiv  Detail & Related papers  (2026-04-30T06:38:28Z)
• Closing the Distribution Gap in Adversarial Training for LLMs [50.33186122381395]
  Adversarial training for LLMs is one of the most promising methods to reliably improve robustness against adversaries.<n>We argue that current adversarial training algorithms minimize adversarial loss on their training set but inadequately cover the data distribution, resulting in vulnerability to seemingly simple attacks.<n>We propose Distributional Adversarial Training, DAT, to approximate the true joint distribution of prompts and responses.
  arXiv  Detail & Related papers  (2026-02-16T22:34:52Z)
• Deep Leakage with Generative Flow Matching Denoiser [54.05993847488204]
  We introduce a new deep leakage (DL) attack that integrates a generative Flow Matching (FM) prior into the reconstruction process.<n>Our approach consistently outperforms state-of-the-art attacks across pixel-level, perceptual, and feature-based similarity metrics.
  arXiv  Detail & Related papers  (2026-01-21T14:51:01Z)
• Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal [28.588188876688037]
  Pre-trained language models (PLMs) have driven substantial progress in natural language processing but remain vulnerable to adversarial attacks.<n>We propose a simple yet effective add-on module that enhances the adversarial robustness of PLMs.
  arXiv  Detail & Related papers  (2025-07-29T12:31:26Z)
• Adversarial Training in Low-Label Regimes with Margin-Based Interpolation [8.585017175426023]
  Adversarial training has emerged as an effective approach to train robust neural network models that are resistant to adversarial attacks.<n>In this paper, we introduce a novel semi-supervised adversarial training approach that enhances both robustness and natural accuracy.
  arXiv  Detail & Related papers  (2024-11-27T00:35:13Z)
• Efficient Adversarial Training in LLMs with Continuous Attacks [99.5882845458567]
  Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. We propose a fast adversarial training algorithm (C-AdvUL) composed of two losses. C-AdvIPO is an adversarial variant of IPO that does not require utility data for adversarially robust alignment.
  arXiv  Detail & Related papers  (2024-05-24T14:20:09Z)
• Doubly Robust Instance-Reweighted Adversarial Training [107.40683655362285]
  We propose a novel doubly-robust instance reweighted adversarial framework. Our importance weights are obtained by optimizing the KL-divergence regularized loss function. Our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance.
  arXiv  Detail & Related papers  (2023-08-01T06:16:18Z)
• On the Generalization Properties of Adversarial Training [21.79888306754263]
  This paper studies the generalization performance of a generic adversarial training algorithm. A series of numerical studies are conducted to demonstrate how the smoothness and L1 penalization help improve the adversarial robustness of models.
  arXiv  Detail & Related papers  (2020-08-15T02:32:09Z)
• Adversarial Distributional Training for Robust Deep Learning [53.300984501078126]
  Adversarial training (AT) is among the most effective techniques to improve model robustness by augmenting training data with adversarial examples. Most existing AT methods adopt a specific attack to craft adversarial examples, leading to the unreliable robustness against other unseen attacks. In this paper, we introduce adversarial distributional training (ADT), a novel framework for learning robust models.
  arXiv  Detail & Related papers  (2020-02-14T12:36:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.