Speculative Safety-Aware Decoding

• URL: http://arxiv.org/abs/2508.17739v2
• Date: Sun, 28 Sep 2025 05:23:45 GMT
• Title: Speculative Safety-Aware Decoding
• Authors: Xuekang Wang, Shengyu Zhu, Xueqi Cheng,
• Abstract summary: We introduce Speculative Safety-Aware Decoding (SSD), a lightweight decoding-time approach that equips LLMs with the desired safety property while accelerating inference.<n>SSD integrates speculative sampling during decoding and leverages the match ratio between the small and composite models to quantify jailbreak risks.<n> Experimental results show that SSD successfully equips the large model with the desired safety property, and also allows the model to remain helpful to benign queries.
• Score: 46.78651034593231
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite extensive efforts to align Large Language Models (LLMs) with human values and safety rules, jailbreak attacks that exploit certain vulnerabilities continuously emerge, highlighting the need to strengthen existing LLMs with additional safety properties to defend against these attacks. However, tuning large models has become increasingly resource intensive and may have difficulty ensuring consistent performance. We introduce Speculative Safety-Aware Decoding (SSD), a lightweight decoding-time approach that equips LLMs with the desired safety property while accelerating inference. We assume that there exists a small language model that possesses this desired property. SSD integrates speculative sampling during decoding and leverages the match ratio between the small and composite models to quantify jailbreak risks. This enables SSD to dynamically switch between decoding schemes to prioritize utility or safety, to handle the challenge of different model capacities. The output token is then sampled from a new distribution that combines the distributions of the original and the small models. Experimental results show that SSD successfully equips the large model with the desired safety property, and also allows the model to remain helpful to benign queries. Furthermore, SSD accelerates the inference time, thanks to the speculative sampling design.

Related papers

• Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility [26.564913442069866]
  Vision language models (VLMs) extend the reasoning capabilities of large language models (LLMs) to cross-modal settings.<n>Existing defenses rely on safety fine-tuning or aggressive token manipulations, incurring substantial training costs or significantly degrading utility.<n>We propose Risk Awareness Injection (RAI), a lightweight and training-free framework for safety calibration.
  arXiv  Detail & Related papers  (2026-02-03T11:26:05Z)
• Patching LLM Like Software: A Lightweight Method for Improving Safety Policy in Large Language Models [63.54707418559388]
  We propose patching for large language models (LLMs) like software versions.<n>Our method enables rapid remediation by prepending a compact, learnable prefix to an existing model.
  arXiv  Detail & Related papers  (2025-11-11T17:25:44Z)
• Reimagining Safety Alignment with An Image [49.33281424100804]
  Large language models (LLMs) excel in diverse applications but face dual challenges: generating harmful content under jailbreak attacks and over-refusal of benign queries.<n>We propose Magic Image, an optimization-driven visual prompt framework that enhances security while reducing over-refusal.
  arXiv  Detail & Related papers  (2025-11-01T11:27:07Z)
• RAID: Refusal-Aware and Integrated Decoding for Jailbreaking LLMs [17.313975711973374]
  RAID (Refusal-Aware and Integrated Decoding) is a framework that crafts adversarial suffixes that induce restricted content while preserving fluency.<n>We show that RAID achieves higher attack success rates with fewer queries and lower computational cost than recent white-box and black-box baselines.
  arXiv  Detail & Related papers  (2025-10-14T19:33:09Z)
• Self Speculative Decoding for Diffusion Large Language Models [21.955478721386953]
  Diffusion-based Large Language Models (dLLMs) have emerged as a competitive alternative to autoregressive models.<n>We propose textbfSelf textbfSpeculative textbfDecoding (SSD) to leverage the dLLM itself as both speculative decoding drafter and verifier.<n>SSD achieves up to 3.46$times$ speedup while keeping the output identical to stepwise decoding on open source models such as LLaDA and Dream.
  arXiv  Detail & Related papers  (2025-10-05T10:52:28Z)
• SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language Models [59.217270662809696]
  Large Language Models (LLMs) excel at various natural language processing tasks but remain vulnerable to jailbreaking attacks.<n>This paper explores aligning the model's inherent discrimination and generation capabilities.<n>Our method does not require any additional annotated data or external models during the training phase.
  arXiv  Detail & Related papers  (2025-08-21T15:26:09Z)
• Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security [63.41350337821108]
  We propose Secure Tug-of-War (SecTOW) to enhance the security of multimodal large language models (MLLMs)<n>SecTOW consists of two modules: a defender and an auxiliary attacker, both trained iteratively using reinforcement learning (GRPO)<n>We show that SecTOW significantly improves security while preserving general performance.
  arXiv  Detail & Related papers  (2025-07-29T17:39:48Z)
• Jailbreaking Large Language Diffusion Models: Revealing Hidden Safety Flaws in Diffusion-Based Text Generation [7.857304417560443]
  We present a PArallel Decoding jailbreak (PAD) for diffusion-based language models.<n>PAD achieves jailbreak attack success rates by 97%, revealing significant safety vulnerabilities.<n>Compared to autoregressive Large Language Models (LLMs), LLDMs increase the harmful generation speed by 2x.
  arXiv  Detail & Related papers  (2025-07-25T12:53:03Z)
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding [51.711605076319216]
  Diffusion-based large language models (Diffusion LLMs) have shown promise for non-autoregressive text generation with parallel decoding capabilities.<n>We introduce a novel block-wise approximate KV Cache mechanism tailored for bidirectional diffusion models, enabling cache reuse with negligible performance drop.<n>We propose a confidence-aware parallel decoding strategy that selectively decodes tokens exceeding a confidence threshold, mitigating dependency violations and maintaining generation quality.
  arXiv  Detail & Related papers  (2025-05-28T17:39:15Z)
• Robustifying Vision-Language Models via Dynamic Token Reweighting [28.675118345987887]
  Large vision-language models (VLMs) are highly vulnerable to jailbreak attacks.<n>We present a novel inference-time defense that mitigates multimodal jailbreak attacks.<n>We introduce a new formulation of the safety-relevant distributional shift induced by the visual modality.
  arXiv  Detail & Related papers  (2025-05-22T03:00:39Z)
• STShield: Single-Token Sentinel for Real-Time Jailbreak Detection in Large Language Models [31.35788474507371]
  Large Language Models (LLMs) have become increasingly vulnerable to jailbreak attacks.<n>We present STShield, a lightweight framework for real-time jailbroken judgement.
  arXiv  Detail & Related papers  (2025-03-23T04:23:07Z)
• DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing [62.43110639295449]
  Large Language Models (LLMs) are widely applied in decision making, but their deployment is threatened by jailbreak attacks.<n>Delman is a novel approach leveraging direct model editing for precise, dynamic protection against jailbreak attacks.<n>Delman directly updates a minimal set of relevant parameters to neutralize harmful behaviors while preserving the model's utility.
  arXiv  Detail & Related papers  (2025-02-17T10:39:21Z)
• Model-Editing-Based Jailbreak against Safety-aligned Large Language Models [13.887770576598646]
  Large Language Models (LLMs) have transformed numerous fields by enabling advanced natural language interactions.<n>This paper presents Targeted Model Editing (TME), a novel white-box approach that bypasses safety filters.<n>TME identifies and removes safety-critical transformations (SCTs) embedded in model matrices, enabling malicious queries to bypass restrictions.
  arXiv  Detail & Related papers  (2024-12-11T08:44:15Z)
• ProSec: Fortifying Code LLMs with Proactive Security Alignment [14.907702430331803]
  Existing methods collect security-focused datasets from real-world vulnerabilities for instruction tuning.<n>We propose ProSec, a novel proactive security alignment approach designed to align code LLMs with secure coding practices.
  arXiv  Detail & Related papers  (2024-11-19T22:00:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.