Robust Safety Monitoring of Language Models via Activation Watermarking

• URL: http://arxiv.org/abs/2603.23171v2
• Date: Mon, 30 Mar 2026 21:43:49 GMT
• Title: Robust Safety Monitoring of Language Models via Activation Watermarking
• Abstract summary: Large language models (LLMs) can be misused to reveal sensitive information, such as weapon-making instructions or writing malware.<n>LLMs providers rely on $emphmonitoring$ to detect and flag unsafe behavior during inference.<n> Adaptive attackers are a major concern as LLM providers cannot patch their security mechanisms.
• Score: 8.322320917323609
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) can be misused to reveal sensitive information, such as weapon-making instructions or writing malware. LLM providers rely on $\emph{monitoring}$ to detect and flag unsafe behavior during inference. An open security challenge is $\emph{adaptive}$ adversaries who craft attacks that simultaneously (i) evade detection while (ii) eliciting unsafe behavior. Adaptive attackers are a major concern as LLM providers cannot patch their security mechanisms, since they are unaware of how their models are being misused. We cast $\emph{robust}$ LLM monitoring as a security game, where adversaries who know about the monitor try to extract sensitive information, while a provider must accurately detect these adversarial queries at low false positive rates. Our work (i) shows that existing LLM monitors are vulnerable to adaptive attackers and (ii) designs improved defenses through $\emph{activation watermarking}$ by carefully introducing uncertainty for the attacker during inference. We find that $\emph{activation watermarking}$ outperforms guard baselines by up to $52\%$ under adaptive attackers who know the monitoring algorithm but not the secret key.

Related papers

• LMSM: LLM Security Framework Inspired by Linux Security Modules [56.93644694627783]
  Large language models (LLMs) are increasingly deployed with layered defenses, yet malicious prompts can still bypass them.<n>We present Language Model Security Modules (LMSM), a security framework that adapts the separation behind Linux Security Modules (LSM) to LLM serving.
  arXiv  Detail & Related papers  (2026-08-26T12:13:05Z)
• AgentShield: Deception-based Compromise Detection for Tool-using LLM Agents [4.36595961990666]
  AgentShield is a deception-based detection framework.<n>It places three layers of traps inside the agent's tool interface.<n>It catches 90.7%-100% of such successful attacks, with zero false alarms on 485 normal-use tests.
  arXiv  Detail & Related papers  (2026-05-10T20:08:27Z)
• Friend or Foe: How LLMs' Safety Mind Gets Fooled by Intent Shift Attack [53.34204977366491]
  Large language models (LLMs) remain vulnerable to jailbreaking attacks despite their impressive capabilities.<n>In this paper, we introduce ISA (Intent Shift Attack), which obfuscates LLMs about the intent of the attacks.<n>Our approach only needs minimal edits to the original request, and yields natural, human-readable, and seemingly harmless prompts.
  arXiv  Detail & Related papers  (2025-11-01T13:44:42Z)
• $δ$-STEAL: LLM Stealing Attack with Local Differential Privacy [24.88863537562324]
  We introduce $delta$-STEAL, a model stealing attack that bypasses the service provider's watermark detectors while preserving the adversary's model utility.<n>Our experiments show that $delta$-STEAL with lightweight modifications achieves attack success rates of up to $96.95%$ without significantly compromising the adversary's model utility.
  arXiv  Detail & Related papers  (2025-10-24T18:19:38Z)
• Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols [80.68060125494645]
  We study adaptive attacks by an untrusted model that knows the protocol and the monitor model.<n>We instantiate a simple adaptive attack vector by which the attacker embeds publicly known or zero-shot prompt injections in the model outputs.
  arXiv  Detail & Related papers  (2025-10-10T15:12:44Z)
• Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs [95.06033929366203]
  Large language models (LLM) developers aim for their models to be honest, helpful, and harmless.<n>We show that frontier LLMs can develop a preference for dishonesty as a new strategy, even when other options are available.<n>We find no apparent cause for the propensity to deceive, but show that more capable models are better at executing this strategy.
  arXiv  Detail & Related papers  (2025-09-22T17:30:56Z)
• BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attacks [58.959622170433725]
  BlindGuard is an unsupervised defense method that learns without requiring any attack-specific labels or prior knowledge of malicious behaviors.<n>We show that BlindGuard effectively detects diverse attack types (i.e., prompt injection, memory poisoning, and tool attack) across multi-agent systems.
  arXiv  Detail & Related papers  (2025-08-11T16:04:47Z)
• How Not to Detect Prompt Injections with an LLM [19.785755392783287]
  Recent defenses based on $textitknown-answer detection$ (KAD) have achieved near-perfect performance by using an LLM to classify inputs as clean or contaminated.<n>We formally characterize the KAD framework and uncover a structural vulnerability in its design that invalidates its core security premise.<n>We design a methodical adaptive attack, $textitDataFlip$, to exploit this fundamental weakness.
  arXiv  Detail & Related papers  (2025-07-08T03:24:56Z)
• Defeating Prompt Injections by Design [79.00910871948787]
  CaMeL is a robust defense that creates a protective system layer around the Large Language Models.<n>To operate, CaMeL explicitly extracts the control and data flows from the (trusted) query.<n>To further improve security, CaMeL uses a notion of a capability to prevent the exfiltration of private data over unauthorized data flows.
  arXiv  Detail & Related papers  (2025-03-24T15:54:10Z)
• LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attacks [7.013820690538764]
  We study attacks that exploit the emphfalse negatives of safeguard methods.<n>The malicious attackers could also exploit false positives of safeguards, leading to a denial-of-service (DoS) affecting users.
  arXiv  Detail & Related papers  (2024-10-03T19:07:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.