GuardReasoner: Towards Reasoning-based LLM Safeguards

• URL: http://arxiv.org/abs/2501.18492v1
• Date: Thu, 30 Jan 2025 17:06:06 GMT
• Title: GuardReasoner: Towards Reasoning-based LLM Safeguards
• Abstract summary: This paper proposes GuardReasoner, a new safeguard for LLMs.<n>We first create the GuardReasonerTrain dataset, which consists of 127K samples with 460K detailed reasoning steps.<n>Then, we introduce reasoning SFT to unlock the reasoning capability of guard models.<n>In this manner, GuardReasoner achieves better performance, explainability, and generalizability.
• Score: 63.53800124080227
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As LLMs increasingly impact safety-critical applications, ensuring their safety using guardrails remains a key challenge. This paper proposes GuardReasoner, a new safeguard for LLMs, by guiding the guard model to learn to reason. Concretely, we first create the GuardReasonerTrain dataset, which consists of 127K samples with 460K detailed reasoning steps. Then, we introduce reasoning SFT to unlock the reasoning capability of guard models. In addition, we present hard sample DPO to further strengthen their reasoning ability. In this manner, GuardReasoner achieves better performance, explainability, and generalizability. Extensive experiments and analyses on 13 benchmarks of 3 guardrail tasks demonstrate its superiority. Remarkably, GuardReasoner 8B surpasses GPT-4o+CoT by 5.74% and LLaMA Guard 3 8B by 20.84% F1 score on average. We release the training data, code, and models with different scales (1B, 3B, 8B) of GuardReasoner : https://github.com/yueliu1999/GuardReasoner/.

Related papers

• LatentGuard: Efficient and Inspectable Latent Reasoning for LLM Safeguards [21.146106913383868]
  Reasoning-based guard models improve LLM safeguards, but decoding explicit rationales for every interaction makes them costly to deploy.<n>We propose LatentGuard, an efficient and inspectable safeguard framework that brings continuous latent reasoning to guard models.
  arXiv  Detail & Related papers  (2026-08-04T15:47:41Z)
• Robust and Efficient Guardrails with Latent Reasoning [23.837448508316513]
  Reasoning-based guardrails incur substantial query latency and token overhead.<n>We propose COLAGUARD, a guardrail model that transfers multi-step safety reasoning into a continuous latent space.<n>Our results suggest latent reasoning offers a practical alternative to explicit rationale generation for deployable guardrails.
  arXiv  Detail & Related papers  (2026-05-27T20:15:22Z)
• Reflect-Guard: Enhancing LLM Safeguards against Adversarial Prompts via Logical Self-Reflection [5.902327401962825]
  We present Reflect-Guard, a method that augments LLM-based safety classifiers with chain-of-thought self-reflection capabilities.<n>Our approach distills analytical reasoning from GPT-4o-mini into structured annotations, then trains Llama-Guard-3-8B via QLoRA to generate logical self-reflections.<n>Using only 1000 training examples and updating just 0.5% of model parameters, Reflect-Guard achieves substantial improvements on two challenging benchmarks.
  arXiv  Detail & Related papers  (2026-05-24T02:58:21Z)
• Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs [70.81495077853673]
  Hard-gated safety checkers often over-refuse and misalign with a vendor's model spec.<n>This work introduces Guardian-as-an-Advisor (GaaA), a soft-gating pipeline where a guardian predicts a binary risk label and prepends this advice to the original query for re-inference.<n>Overall, GaaA steers models to comply with the model spec, maintaining safety while reducing over-refusal.
  arXiv  Detail & Related papers  (2026-04-08T23:47:29Z)
• Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model [50.29667251847595]
  We show that despite being larger in model size and stronger in safety capability, there exists an alignment gap between teacher and student language models.<n>We propose a BoN sampling method that attributes the unsafe behavior back to the base LLMs in the latent space.<n>In particular, across 7 teacher models and 6 student models of different classes and sizes, we show an average attack success rate (ASR) reduction of 28.2% in DAN, 31.3% in WildJailbreak and 35.4 % in StrongREJECT benchmarks.
  arXiv  Detail & Related papers  (2026-04-01T02:42:41Z)
• Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away [97.11976870616273]
  We propose a lightweight inference-time defense that treats safety recovery as a satising constraint rather than an objective.<n>In our evaluations across six open-source MLRMs and four jailbreak benchmarks, SafeThink reduces attack success rates by 30-60%.
  arXiv  Detail & Related papers  (2026-02-11T18:09:17Z)
• SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models [66.71948519280669]
  Multimodal Large Reasoning Models (MLRMs) demonstrate impressive crossmodal reasoning but often amplify safety risks under adversarial prompts.<n> Existing defenses mainly act at the output level and do not constrain the reasoning process, leaving models to implicit risks.<n>We propose SaFeR-VLM, which integrates four components and supports dynamic and interpretable safety decisions beyond surface-level filtering.
  arXiv  Detail & Related papers  (2025-10-08T10:39:12Z)
• Unleashing the Reasoning Potential of Pre-trained LLMs by Critique Fine-Tuning on One Problem [53.3188041952701]
  We show that Critique Fine-Tuning (CFT) on only one problem can effectively unleash the reasoning potential of LLMs.<n>With just 5 GPU hours of training, Qwen-Math-7B-CFT show an average improvement of 15% on six math benchmarks and 16% on three logic reasoning benchmarks.<n>Results are comparable to or even surpass the results from RL with 20x less compute.
  arXiv  Detail & Related papers  (2025-06-03T18:35:52Z)
• How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study [90.34190170330481]
  Large Reasoning Models (LRMs) have achieved remarkable success on reasoning-intensive tasks such as mathematics and programming.<n>However, their enhanced reasoning capabilities do not necessarily translate to improved safety performance.<n>We present a comprehensive empirical study on how to enhance the safety of LRMs through Supervised Fine-Tuning.
  arXiv  Detail & Related papers  (2025-05-21T11:45:29Z)
• GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning [43.89818154399979]
  This paper introduces a novel reasoning-based VLM guard model dubbed GuardReasoner-VL.<n>We construct GuardReasoner-VLTrain, a reasoning corpus with 123K samples and 631K reasoning steps, spanning text, image, and text-image inputs.<n>To balance performance and token efficiency, we design a length-aware safety reward that integrates accuracy, format, and token cost.
  arXiv  Detail & Related papers  (2025-05-16T09:46:10Z)
• STAR-1: Safer Alignment of Reasoning LLMs with 1K Data [33.51888940162213]
  STAR-1 is a high-quality, just-1k-scale safety dataset specifically designed for large reasoning models (LRMs) Built on three core principles -- diversity, deliberative reasoning, and rigorous filtering -- STAR-1 aims to address the critical needs for safety alignment in LRMs.
  arXiv  Detail & Related papers  (2025-04-02T17:04:04Z)
• DNR Bench: Benchmarking Over-Reasoning in Reasoning LLMs [3.850766603072179]
  We introduce Don't Reason Bench (DNR Bench) to evaluate large language models (LLMs) DNR Bench consists of 150 adversarially designed prompts that are easy for humans to understand and respond to. Our experiments reveal that RLMs generate up to 70x more tokens than necessary, often failing at tasks that simpler non-reasoning models handle efficiently with higher accuracy.
  arXiv  Detail & Related papers  (2025-03-20T02:19:14Z)
• ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrails [33.96886111900147]
  ThinkGuard is a critique-augmented guardrail model that distills knowledge from high-capacity language models. It achieves the highest average F1 and AUPRC, outperforming all baselines. It surpasses label-only fine-tuned models, confirming that structured critiques enhance both classification precision and nuanced safety reasoning.
  arXiv  Detail & Related papers  (2025-02-19T06:09:58Z)
• LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters! [53.84130385074551]
  Large reasoning models (LRMs) tackle complex reasoning problems by following long chain-of-thoughts (Long CoT) We find that a Large Language model (LLM) can effectively learn Long CoT reasoning through data-efficient supervised fine-tuning (SFT) and parameter-efficient low-rank adaptation (LoRA) With just 17k long CoT training samples, the Qwen2.5-32B-Instruct model achieves significant improvements on a wide range of math and coding benchmarks.
  arXiv  Detail & Related papers  (2025-02-11T08:48:48Z)
• Responsible AI in Construction Safety: Systematic Evaluation of Large Language Models and Prompt Engineering [9.559203170987598]
  Construction remains one of the most hazardous sectors. Recent advancements in AI, particularly Large Language Models (LLMs), offer promising opportunities for enhancing workplace safety. This study evaluates the performance of two widely used LLMs, GPT-3.5 and GPT-4o, across three standardized exams administered by the Board of Certified Safety Professionals (BCSP)
  arXiv  Detail & Related papers  (2024-11-13T04:06:09Z)
• Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding [74.31981011985681]
  Large language models (LLMs) have shown impressive capabilities, but still struggle with complex reasoning tasks requiring multiple steps. We introduce LaTent Reasoning Optimization (LaTRO), a principled framework that formulates reasoning as sampling from a latent distribution. We validate LaTRO through experiments on GSM8K and ARC-Challenge datasets using multiple model architectures.
  arXiv  Detail & Related papers  (2024-11-06T22:02:30Z)
• Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training [67.30423823744506]
  This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs) We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position. DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful
  arXiv  Detail & Related papers  (2024-07-12T09:36:33Z)
• $R^2$-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning [8.408258504178718]
  Existing guardrail models treat various safety categories independently and fail to explicitly capture the intercorrelations among them. We propose $R2$-Guard, a robust reasoning enabled LLM guardrail via knowledge-enhanced logical reasoning. $R2$-Guard significantly surpasses SOTA method LlamaGuard by 30.2% on ToxicChat and by 59.5% against jailbreaking attacks.
  arXiv  Detail & Related papers  (2024-07-08T02:15:29Z)
• WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs [54.10865585773691]
  We introduce WildGuard -- an open, light-weight moderation tool for LLM safety. WildGuard achieves three goals: identifying malicious intent in user prompts, detecting safety risks of model responses, and determining model refusal rate.
  arXiv  Detail & Related papers  (2024-06-26T16:58:20Z)
• Revisiting Personalized Federated Learning: Robustness Against Backdoor Attacks [53.81129518924231]
  We conduct the first study of backdoor attacks in the pFL framework. We show that pFL methods with partial model-sharing can significantly boost robustness against backdoor attacks. We propose a lightweight defense method, Simple-Tuning, which empirically improves defense performance against backdoor attacks.
  arXiv  Detail & Related papers  (2023-02-03T11:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.