The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play

• URL: http://arxiv.org/abs/2605.08427v1
• Date: Fri, 08 May 2026 19:41:59 GMT
• Title: The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play
• Abstract summary: Self-play red team is an established approach to improving AI safety.<n>We propose Anchored Bipolicy Self-Play, which trains role-specific LoRA adapters on top of a frozen base model.<n>We show up to 100x greater parameter efficiency than finetuning and consistent improvements in safety compared to self-play fine-tuned models.
• Score: 16.696570190611112
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Self-play red team is an established approach to improving AI safety in which different instances of the same model play attacker and defender roles in a zero-sum game, i.e., where the attacker tries to jailbreak the defender; if self-play converges to a Nash equilibrium, the model is guaranteed to respond safely within the settings of the game. Although the parameter sharing enforced by the use of the same model for the two roles improves stability and performance, it introduces fundamental theoretical and architectural limitations. We show that the set of Nash equilibria that can be reached corresponds to a broad class of behaviours that includes trivial always refuse strategies and oracle-like defenders, thus limiting practical applicability. We then show that when attacker and defender share and update the same base model, the dynamics collapse to self-consistency, so that attacks do not enforce adversarial pressure on the defender. In response, we propose Anchored Bipolicy Self-Play, which trains distinct role-specific LoRA adapters on top of a frozen base model, thereby maintaining stable optimisation while preserving adversarial pressure through explicit role separation. In relation to standard self-play, we show up to 100x greater parameter efficiency than finetuning and consistent improvements in safety compared to self-play fine-tuned models. We evaluate on Qwen2.5-{3B, 7B,14B}-IT models across widely used safety benchmarks, showing improved robustness without loss of reasoning ability. Cross-play experiments further show that our attacker and defender models are superior to self-play in terms of adversarial defence and safety.

Related papers

• On the Vulnerability of Parameter-Level Defenses to Model Merging [75.62810606927121]
  We propose the Anchor-Guided Attack (AGA) to circumvent existing safeguards.<n>AGA consistently bypasses both individual and composite defenses under realistic defense-agnostic scenarios.<n>We provide Anchor-Repulsive Fine-tuning (ARF) to mitigate the anchor dominance leveraged by AGA.
  arXiv  Detail & Related papers  (2026-06-29T14:26:13Z)
• Reproducing AlphaZero on Tablut: Self-Play RL for an Asymmetric Board Game [0.6574517227976925]
  This work investigates the adaptation of the AlphaZero reinforcement learning algorithm to Tablut, an asymmetric historical board game.<n>Over 100 self-play iterations, the modified model demonstrated steady improvement, achieving a BayesElo rating of 1235.<n>Ultimately, the experiments confirm that AlphaZero's self-play framework can transfer to highly asymmetric games.
  arXiv  Detail & Related papers  (2026-04-07T06:16:19Z)
• ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Driving [63.980630608984605]
  This paper presents ADV-0, a closed-loop min-max optimization framework that treats the interaction between driving policy (defender) and adversarial agent (attacker) as a zero-sum Markov game.<n>To make this tractable, we cast dynamic adversary evolution as iterative preference learning, efficiently approximating this optimum and offering an algorithm-agnostic solution to the game.<n> Experiments indicate that it effectively exposes diverse safety-critical failures and greatly enhances the generalizability of both learned policies and motion planners against unseen long-tail risks.
  arXiv  Detail & Related papers  (2026-03-16T12:58:31Z)
• MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safety [28.246225272659917]
  This paper introduces textbfMAGIC, a novel multi-turn multi-agent reinforcement learning framework.<n>It formulates Large Language Models safety alignment as an adversarial asymmetric game.<n>Our framework demonstrates superior defense success rates without compromising the helpfulness of the model.
  arXiv  Detail & Related papers  (2026-02-02T02:12:28Z)
• Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replay [19.431152130507648]
  Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to adversarial jailbreak'' attacks.<n>This paper introduces Safety Self- Play (SSP), a system that acts as both the Attacker (generating jailbreaks) and the Defender (refusing harmful requests)<n>SSP autonomously evolves robust defense capabilities, significantly outperforming baselines trained on static adversarial datasets.
  arXiv  Detail & Related papers  (2026-01-15T17:00:16Z)
• Safety Alignment of LMs via Non-cooperative Games [51.83432183158595]
  Current approaches rely on sequential adversarial training, generating adversarial prompts and fine-tuning LMs to defend against them.<n>We introduce a different paradigm: framing safety alignment as a non-zero-sum game between an Attacker LM and a Defender LM trained jointly.<n>Our method uses a preference-based reward signal derived from pairwise comparisons instead of point-wise scores, providing more robust supervision and potentially reducing reward hacking.
  arXiv  Detail & Related papers  (2025-12-23T22:13:14Z)
• Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models [64.47869632167284]
  Conventional language model (LM) safety alignment relies on a reactive, disjoint procedure: attackers exploit a static model, followed by defensive fine-tuning to patch exposed vulnerabilities.<n>This sequential approach creates a mismatch -- attackers overfit to obsolete defenses, while defenders perpetually lag behind emerging threats.<n>We propose Self-RedTeam, an online self-play reinforcement learning algorithm where an attacker and defender agent co-evolve through continuous interaction.
  arXiv  Detail & Related papers  (2025-06-09T06:35:12Z)
• Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularization [53.04697800214848]
  Adversarial Training (AT) is pivotal in fortifying the robustness of deep learning models. AT methods, relying on direct iterative updates for target model's defense, frequently encounter obstacles such as unstable training and catastrophic overfitting. We present a general proxy guided defense framework, LAST' (bf Learn from the Pbf ast)
  arXiv  Detail & Related papers  (2023-10-19T13:13:41Z)
• MultiRobustBench: Benchmarking Robustness Against Multiple Attacks [86.70417016955459]
  We present the first unified framework for considering multiple attacks against machine learning (ML) models. Our framework is able to model different levels of learner's knowledge about the test-time adversary. We evaluate the performance of 16 defended models for robustness against a set of 9 different attack types.
  arXiv  Detail & Related papers  (2023-02-21T20:26:39Z)
• Adversarial Machine Learning and Defense Game for NextG Signal Classification with Deep Learning [1.1726528038065764]
  NextG systems can employ deep neural networks (DNNs) for various tasks such as user equipment identification, physical layer authentication, and detection of incumbent users. This paper presents a game-theoretic framework to study the interactions of attack and defense for deep learning-based NextG signal classification.
  arXiv  Detail & Related papers  (2022-12-22T15:13:03Z)
• Adaptive Feature Alignment for Adversarial Training [56.17654691470554]
  CNNs are typically vulnerable to adversarial attacks, which pose a threat to security-sensitive applications. We propose the adaptive feature alignment (AFA) to generate features of arbitrary attacking strengths. Our method is trained to automatically align features of arbitrary attacking strength.
  arXiv  Detail & Related papers  (2021-05-31T17:01:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.