$R^2$-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning

• URL: http://arxiv.org/abs/2407.05557v1
• Date: Mon, 8 Jul 2024 02:15:29 GMT
• Title: $R^2$-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning
• Authors: Mintong Kang, Bo Li,
• Abstract summary: Existing guardrail models treat various safety categories independently and fail to explicitly capture the intercorrelations among them. We propose $R2$-Guard, a robust reasoning enabled LLM guardrail via knowledge-enhanced logical reasoning. $R2$-Guard significantly surpasses SOTA method LlamaGuard by 30.2% on ToxicChat and by 59.5% against jailbreaking attacks.
• Score: 8.408258504178718
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As LLMs become increasingly prevalent across various applications, it is critical to establish safety guardrails to moderate input/output content of LLMs. Existing guardrail models treat various safety categories independently and fail to explicitly capture the intercorrelations among them. This has led to limitations such as ineffectiveness due to inadequate training on long-tail data from correlated safety categories, susceptibility to jailbreaking attacks, and inflexibility regarding new safety categories. To address these limitations, we propose $R^2$-Guard, a robust reasoning enabled LLM guardrail via knowledge-enhanced logical reasoning. Specifically, $R^2$-Guard comprises two parts: data-driven category-specific learning and reasoning components. The data-driven guardrail models provide unsafety probabilities of moderated content on different safety categories. We then encode safety knowledge among different categories as first-order logical rules and embed them into a probabilistic graphic model (PGM) based reasoning component. The unsafety probabilities of different categories from data-driven guardrail models are sent to the reasoning component for final inference. We employ two types of PGMs: Markov logic networks (MLNs) and probabilistic circuits (PCs), and optimize PCs to achieve precision-efficiency balance via improved graph structure. To further perform stress tests for guardrail models, we employ a pairwise construction method to construct a new safety benchmark TwinSafety, which features principled categories. We demonstrate the effectiveness of $R^2$-Guard by comparisons with eight strong guardrail models on six safety benchmarks, and demonstrate the robustness of $R^2$-Guard against four SOTA jailbreaking attacks. $R^2$-Guard significantly surpasses SOTA method LlamaGuard by 30.2% on ToxicChat and by 59.5% against jailbreaking attacks.

Related papers

• ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrails [33.96886111900147]
  ThinkGuard is a critique-augmented guardrail model that distills knowledge from high-capacity language models. It achieves the highest average F1 and AUPRC, outperforming all baselines. It surpasses label-only fine-tuned models, confirming that structured critiques enhance both classification precision and nuanced safety reasoning.
  arXiv  Detail & Related papers  (2025-02-19T06:09:58Z)
• SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities [21.317245896641136]
  Long chain-of-thought (CoT) reasoning generates structured intermediate steps, enhancing reasoning capabilities. Current research on large language model (LLM) safety usually focuses on short-answer responses, overlooking the long CoT style outputs of LRMs.
  arXiv  Detail & Related papers  (2025-02-17T16:57:56Z)
• GuardReasoner: Towards Reasoning-based LLM Safeguards [63.53800124080227]
  This paper proposes GuardReasoner, a new safeguard for LLMs. We first create the GuardReasonerTrain dataset, which consists of 127K samples with 460K detailed reasoning steps. Then, we introduce reasoning SFT to unlock the reasoning capability of guard models. In this manner, GuardReasoner achieves better performance, explainability, and generalizability.
  arXiv  Detail & Related papers  (2025-01-30T17:06:06Z)
• HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models [92.85175340702125]
  We distill a large teacher safety guard model into a smaller one using a labeled dataset of instruction-response pairs with binary harmfulness labels. We propose HarmAug, a simple yet effective data augmentation method that involves jailbreaking an LLM and prompting it to generate harmful instructions. Our HarmAug achieves an F1 score comparable to larger models with over 7 billion parameters, and even outperforms them in AUPRC, while operating at less than 25% of their computational cost.
  arXiv  Detail & Related papers  (2024-10-02T13:12:13Z)
• PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routing [1.474945380093949]
  Inference-Time Guardrails (ITG) offer solutions that shift model output distributions towards compliance. Current methods struggle in balancing safety with helpfulness. We propose PrimeGuard, a novel ITG method that utilizes structured control flow.
  arXiv  Detail & Related papers  (2024-07-23T09:14:27Z)
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study [64.9691741899956]
  Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. We design a synthetic data generation framework that captures salient aspects of an unsafe input. Using this, we investigate three well-known safety fine-tuning methods.
  arXiv  Detail & Related papers  (2024-07-14T16:12:57Z)
• Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training [67.30423823744506]
  This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs) We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position. DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful
  arXiv  Detail & Related papers  (2024-07-12T09:36:33Z)
• WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs [54.10865585773691]
  We introduce WildGuard -- an open, light-weight moderation tool for LLM safety. WildGuard achieves three goals: identifying malicious intent in user prompts, detecting safety risks of model responses, and determining model refusal rate.
  arXiv  Detail & Related papers  (2024-06-26T16:58:20Z)
• SafeAligner: Safety Alignment against Jailbreak Attacks via Response Disparity Guidance [48.36220909956064]
  SafeAligner is a methodology implemented at the decoding stage to fortify defenses against jailbreak attacks. We develop two specialized models: the Sentinel Model, which is trained to foster safety, and the Intruder Model, designed to generate riskier responses. We show that SafeAligner can increase the likelihood of beneficial tokens, while reducing the occurrence of harmful ones.
  arXiv  Detail & Related papers  (2024-06-26T07:15:44Z)
• Safe Linear Bandits over Unknown Polytopes [39.177982674455784]
  The safe linear bandit problem (SLB) is an online approach to linear programming with unknown objective and unknown roundwise constraints. We study the tradeoffs between efficacy and smooth safety costs of SLBs over polytopes.
  arXiv  Detail & Related papers  (2022-09-27T21:13:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.