kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail

• URL: http://arxiv.org/abs/2607.02072v2
• Date: Mon, 06 Jul 2026 14:17:36 GMT
• Title: kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail
• Abstract summary: kNNGuard is a training-free guardrail that utilizes the activation space of an off-the-shelf LLM.<n>It runs 2.7x faster than the best comparable guardrail, and 10x faster than a fine-tuned safety classifier without gradient updates or fine-tuning.
• Score: 1.4914567984930789
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) are increasingly deployed in domains requiring guardrails to detect unsafe, off-topic, or adversarial prompts. Existing guardrails predominantly rely on fine-tuning to build classifiers, which often suffer from low generalization and high inference latency. We present kNNGuard, a training-free guardrail that utilizes the activation space of an off-the-shelf LLM. Given a small bank of 50 safe and unsafe prompts, kNNGuard extracts hidden activations and performs multi-layer kNN fusing activation-space and embedding-space scores for classification. Across six domains spanning topical and security prompts, kNNGuard achieves competitive or superior F1 compared to fine-tuned state-of-the-art guardrails while running 2.7x faster than the best comparable guardrail, and 10x faster than a fine-tuned safety classifier without gradient updates or fine-tuning. Domain adaptation requires only updating the labeled bank, which can be constructed in under 10 seconds and several orders of magnitude faster than established guardrails. We also analyze the impact of system prompts, layer selection, and integration into production LLM pipelines as a configurable, low-latency guardrail.

Related papers

• Privacy-Preserving Topology-Guided Safety for LLM-Based Multi-Agent Systems via Federated Graph Learning [67.04448659688579]
  FGLGuard cuts AgentDojo's ground-success rate by 43% at near-unguarded utility, zero API cost, and negligible capability loss.<n>Federation is not optional: off-the-shelf transfer collapses under distribution shift.<n>Live FGLGuard cuts AgentDojo's ground-success rate by 43% at near-unguarded utility, zero API cost, and negligible capability loss.
  arXiv  Detail & Related papers  (2026-09-02T07:57:12Z)
• StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety-Utility Balancing [80.39644823192113]
  We propose StepGuard, a step-level guard model that can audit completed agent trajectories and check tool actions before they are executed.<n>StepGuard achieves the highest average accuracy among open-weight guard models, with performance comparable to GPT-5.4.<n>When used to guard agents on AgentDojo and AgentDyn, StepGuard reduces mean attack success rate by 77.3%.
  arXiv  Detail & Related papers  (2026-08-25T16:17:27Z)
• DT-Guard: Intent-Driven Reasoning-Active Training for Reasoning-Free LLM Safety Guardrail [19.59614205209542]
  We present DT-Guard, a content safety guardrail model based on a Reasoning-Active Training, Reasoning-Free Inference paradigm.<n>The key idea is to use reasoning supervision during training while emitting only structured safety labels at inference time.
  arXiv  Detail & Related papers  (2026-07-07T14:25:05Z)
• Behind the Refusal: Determining Guardrail Activation via Behavioral Monitoring [1.1391142149008149]
  Guardrail systems that detect and block malicious instructions are an essential component of AI security.<n>We propose the first black-box guardrail reconnaissance methodology, which detects the presence of a guardrail within a target AI system.
  arXiv  Detail & Related papers  (2026-07-02T12:59:28Z)
• BraveGuard: From Open-World Threats to Safer Computer-Use Agents [68.38397514360227]
  BraveGuard is a self-evolving defense framework for training guard models from open-world threat signals and realistic agent trajectories.<n>We instantiate BraveGuard by training multiple guard backbones, including Qwen3-Guard and Llama-Guard variants, and evaluate the resulting guards on trajectory-level agent-safety benchmarks.
  arXiv  Detail & Related papers  (2026-05-31T11:16:18Z)
• Prompt Overflow: What the Guardrail Inspects Is Not What the Model Infers [22.77036856603702]
  Guardrail models are widely deployed to screen user inputs before they reach large language models (LLMs)<n>We introduce a novel Prompt Overflow Attack, which exploits a mismatch between the limited inspection windows of guardrail models and the substantially larger context inference windows of downstream LLMs.<n>We demonstrate that prompts reliably detected in short-context settings can evade guardrail models once adversarially manipulated into over-length inputs.
  arXiv  Detail & Related papers  (2026-05-22T03:27:18Z)
• The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search [58.8834056209347]
  Large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safety guardrails to elicit harmful outputs.<n>We introduce the Correlated Knowledge Attack Agent (CKA-Agent), a dynamic framework that reframes jailbreaking as an adaptive, tree-structured exploration of the target model's knowledge base.
  arXiv  Detail & Related papers  (2025-12-01T07:05:23Z)
• AdaptiveGuard: Towards Adaptive Runtime Safety for LLM-Powered Software [11.606665113249298]
  Guardrails are critical for the safe deployment of Large Language Models (LLMs)-powered software.<n>We propose AdaptiveGuard, an adaptive guardrail that detects novel jailbreak attacks as out-of-distribution (OOD) inputs.<n>We show that AdaptiveGuard achieves 96% OOD detection accuracy, adapts to new attacks in just two update steps, and retains over 85% F1-score on in-distribution data post-adaptation.
  arXiv  Detail & Related papers  (2025-09-21T01:22:42Z)
• SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism [123.54980913741828]
  Multimodal Large Language Models (MLLMs) extend LLMs to support visual reasoning.<n>MLLMs are susceptible to multimodal jailbreak attacks and hindering their safe deployment.<n>We propose Safe Prune-then-Restore (SafePTR), a training-free defense framework that selectively prunes harmful tokens at vulnerable layers while restoring benign features at subsequent layers.
  arXiv  Detail & Related papers  (2025-07-02T09:22:03Z)
• No Free Lunch with Guardrails [0.0]
  We evaluate whether current guardrails effectively prevent misuse while maintaining practical utility.<n>Our findings confirm that there is no free lunch with guardrails; strengthening security often comes at the cost of usability.<n>We propose a blueprint for designing better guardrails that minimize risk while maintaining usability.
  arXiv  Detail & Related papers  (2025-04-01T05:46:54Z)
• Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attacks [59.300698230887114]
  Large language models (LLMs) are shown to be vulnerable to jailbreaking attacks where adversarial prompts are designed to elicit harmful responses.<n>We propose a safety steering framework grounded in safe control theory, ensuring invariant safety in multi-turn dialogues.
  arXiv  Detail & Related papers  (2025-02-28T21:10:03Z)
• CoreGuard: Safeguarding Foundational Capabilities of LLMs Against Model Stealing in Edge Deployment [66.72332011814183]
  CoreGuard is a computation- and communication-efficient protection method for proprietary large language models (LLMs) deployed on edge devices.<n> CoreGuard employs an efficient protection protocol to reduce computational overhead and minimize communication overhead via a propagation protocol.
  arXiv  Detail & Related papers  (2024-10-16T08:14:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.