Can Safety Fine-Tuning Be More Principled? Lessons Learned from Cybersecurity

• URL: http://arxiv.org/abs/2501.11183v1
• Date: Sun, 19 Jan 2025 21:49:42 GMT
• Title: Can Safety Fine-Tuning Be More Principled? Lessons Learned from Cybersecurity
• Abstract summary: We argue that current safety fine-tuning is very similar to a traditional cat-and-mouse game between attackers and defenders in cybersecurity.<n>We show how current defenses are insufficient to prevent new adversarial jailbreak attacks, reward hacking, and loss of control problems.
• Score: 46.83576558654259
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As LLMs develop increasingly advanced capabilities, there is an increased need to minimize the harm that could be caused to society by certain model outputs; hence, most LLMs have safety guardrails added, for example via fine-tuning. In this paper, we argue the position that current safety fine-tuning is very similar to a traditional cat-and-mouse game (or arms race) between attackers and defenders in cybersecurity. Model jailbreaks and attacks are patched with bandaids to target the specific attack mechanism, but many similar attack vectors might remain. When defenders are not proactively coming up with principled mechanisms, it becomes very easy for attackers to sidestep any new defenses. We show how current defenses are insufficient to prevent new adversarial jailbreak attacks, reward hacking, and loss of control problems. In order to learn from past mistakes in cybersecurity, we draw analogies with historical examples and develop lessons learned that can be applied to LLM safety. These arguments support the need for new and more principled approaches to designing safe models, which are architected for security from the beginning. We describe several such approaches from the AI literature.

Related papers

• One Step to the Side: Why Defenses Against Malicious Finetuning Fail Under Adaptive Adversaries [2.657356138730743]
  Model providers increasingly release open weights or allow users to fine-tune foundation models through APIs.<n>Recent defenses aim to make models robust to such malicious fine-tuning, but they are largely evaluated only against fixed attacks that do not account for the defense.<n>We show that these robustness claims are incomplete.<n> Surveying 15 recent defenses, we identify several defense mechanisms and show that they share a single weakness: they obscure or misdirect the path to harmful behavior without removing the behavior itself.<n>We then develop a unified adaptive attack that breaks defenses across all defense mechanisms.
  arXiv  Detail & Related papers  (2026-05-14T09:22:14Z)
• MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safety [28.246225272659917]
  This paper introduces textbfMAGIC, a novel multi-turn multi-agent reinforcement learning framework.<n>It formulates Large Language Models safety alignment as an adversarial asymmetric game.<n>Our framework demonstrates superior defense success rates without compromising the helpfulness of the model.
  arXiv  Detail & Related papers  (2026-02-02T02:12:28Z)
• Friend or Foe: How LLMs' Safety Mind Gets Fooled by Intent Shift Attack [53.34204977366491]
  Large language models (LLMs) remain vulnerable to jailbreaking attacks despite their impressive capabilities.<n>In this paper, we introduce ISA (Intent Shift Attack), which obfuscates LLMs about the intent of the attacks.<n>Our approach only needs minimal edits to the original request, and yields natural, human-readable, and seemingly harmless prompts.
  arXiv  Detail & Related papers  (2025-11-01T13:44:42Z)
• Between a Rock and a Hard Place: Exploiting Ethical Reasoning to Jailbreak LLMs [19.15237483387352]
  We introduce TRIAL (Trolley-problem Reasoning for Interactive Attack Logic), a framework that leverages LLMs ethical reasoning to bypass their safeguards.<n> TRIAL demonstrates high jailbreak success rates towards both open and close-source models.<n>Our findings underscore a fundamental limitation in AI safety: as models gain advanced reasoning abilities, the nature of their alignment may inadvertently allow for more covert security vulnerabilities to be exploited.
  arXiv  Detail & Related papers  (2025-09-04T05:53:20Z)
• Defending Against Prompt Injection With a Few DefensiveTokens [53.7493897456957]
  Large language model (LLM) systems interact with external data to perform complex tasks.<n>By injecting instructions into the data accessed by the system, an attacker can override the initial user task with an arbitrary task directed by the attacker.<n>Test-time defenses, e.g., defensive prompting, have been proposed for system developers to attain security only when needed in a flexible manner.<n>We propose DefensiveToken, a test-time defense with prompt injection comparable to training-time alternatives.
  arXiv  Detail & Related papers  (2025-07-10T17:51:05Z)
• Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval [25.17143802138141]
  Large Language Models (LLMs) are vulnerable to jailbreaking attacks, wherein adversaries exploit carefully engineered prompts to induce harmful or unethical responses.<n>We propose Safety Context Retrieval (SCR), a scalable and robust safeguarding paradigm for LLMs against jailbreaking.
  arXiv  Detail & Related papers  (2025-05-21T16:58:14Z)
• Dark LLMs: The Growing Threat of Unaligned AI Models [8.183446952097528]
  Large Language Models (LLMs) rapidly reshape modern life, advancing fields from healthcare to education and beyond.<n>The vulnerability of LLMs to jailbreak attacks stems from the very data they learn from.<n>Our research identifies the growing threat posed by dark LLMs models deliberately designed without ethical guardrails or modified through jailbreak techniques.
  arXiv  Detail & Related papers  (2025-05-15T08:07:04Z)
• LightDefense: A Lightweight Uncertainty-Driven Defense against Jailbreaks via Shifted Token Distribution [84.2846064139183]
  Large Language Models (LLMs) face threats from jailbreak prompts. We propose LightDefense, a lightweight defense mechanism targeted at white-box models.
  arXiv  Detail & Related papers  (2025-04-02T09:21:26Z)
• Jailbreaking Large Language Models in Infinitely Many Ways [3.5674816606221182]
  We show how one can bypass the safeguards of the most powerful open- and closed-source LLMs and generate content that explicitly violates their safety policies.<n>For two categories of attacks that are straightforward to implement, we discuss two defensive strategies, one in token and the other in embedding space.
  arXiv  Detail & Related papers  (2025-01-18T15:39:53Z)
• Layer-Level Self-Exposure and Patch: Affirmative Token Mitigation for Jailbreak Attack Defense [55.77152277982117]
  We introduce Layer-AdvPatcher, a methodology designed to defend against jailbreak attacks. We use an unlearning strategy to patch specific layers within large language models through self-augmented datasets. Our framework reduces the harmfulness and attack success rate of jailbreak attacks.
  arXiv  Detail & Related papers  (2025-01-05T19:06:03Z)
• Taking off the Rose-Tinted Glasses: A Critical Look at Adversarial ML Through the Lens of Evasion Attacks [11.830908033835728]
  We argue that overly permissive attack and overly restrictive defensive threat models have hampered defense development in the ML domain. We analyze adversarial machine learning from a system security perspective rather than an AI perspective.
  arXiv  Detail & Related papers  (2024-10-15T21:33:23Z)
• Purple-teaming LLMs with Adversarial Defender Training [57.535241000787416]
  We present Purple-teaming LLMs with Adversarial Defender training (PAD) PAD is a pipeline designed to safeguard LLMs by novelly incorporating the red-teaming (attack) and blue-teaming (safety training) techniques. PAD significantly outperforms existing baselines in both finding effective attacks and establishing a robust safe guardrail.
  arXiv  Detail & Related papers  (2024-07-01T23:25:30Z)
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep [48.823599143711235]
  The safety alignment of current Large Language Models (LLMs) is vulnerable. Relatively simple attacks, or even benign fine-tuning, can jailbreak aligned models. We show that deepening the safety alignment beyond just the first few tokens can often meaningfully improve robustness against some common exploits.
  arXiv  Detail & Related papers  (2024-06-10T00:35:23Z)
• Coercing LLMs to do and reveal (almost) anything [80.8601180293558]
  It has been shown that adversarial attacks on large language models (LLMs) can "jailbreak" the model into making harmful statements. We argue that the spectrum of adversarial attacks on LLMs is much larger than merely jailbreaking.
  arXiv  Detail & Related papers  (2024-02-21T18:59:13Z)
• Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignment [31.24530091590395]
  We study an attack scenario called Trojan Activation Attack (TA2), which injects trojan steering vectors into the activation layers of Large Language Models. Our experiment results show that TA2 is highly effective and adds little or no overhead to attack efficiency.
  arXiv  Detail & Related papers  (2023-11-15T23:07:40Z)
• Jailbroken: How Does LLM Safety Training Fail? [92.8748773632051]
  "jailbreak" attacks on early releases of ChatGPT elicit undesired behavior. We investigate why such attacks succeed and how they can be created. New attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests.
  arXiv  Detail & Related papers  (2023-07-05T17:58:10Z)
• Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AI-NIDS [70.60975663021952]
  We study blackbox adversarial attacks on network classifiers. We argue that attacker-defender fixed points are themselves general-sum games with complex phase transitions. We show that a continual learning approach is required to study attacker-defender dynamics.
  arXiv  Detail & Related papers  (2021-11-23T23:42:16Z)
• Arms Race in Adversarial Malware Detection: A Survey [33.8941961394801]
  Malicious software (malware) is a major cyber threat that has to be tackled with Machine Learning (ML) techniques. ML is vulnerable to attacks known as adversarial examples. Knowing the defender's feature set is critical to the success of transfer attacks. The effectiveness of adversarial training depends on the defender's capability in identifying the most powerful attack.
  arXiv  Detail & Related papers  (2020-05-24T07:20:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.