Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation

• URL: http://arxiv.org/abs/2605.28830v1
• Date: Fri, 10 Apr 2026 06:55:07 GMT
• Title: Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation
• Abstract summary: We evaluate 14 open-source safety guard models on a curated benchmark of 79,331 samples spanning 8 NIST AI Risk Framework safety categories.<n>We find that recall is the critical metric for safety applications, as missing harmful content poses greater risk than false positives.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As Large Language Models (LLMs) are increasingly deployed in safety-critical applications, robust content moderation becomes essential. We present a comprehensive evaluation of 14 open-source safety guard models on a curated benchmark of 79,331 samples spanning 8 NIST AI Risk Framework safety categories. Our benchmark aggregates four diverse datasets (HarmBench, StrongREJECT, RealToxicityPrompts, and BeaverTails), filtered to focus exclusively on safety-relevant content (violence, hate speech, harassment, sexual content, suicide/self-harm, profanity, threats, and health misinformation). We find that recall is the critical metric for safety applications, as missing harmful content poses greater risk than false positives. Our evaluation reveals surprising results: Qwen Guard (4B parameters) achieves the highest recall (83.97%) while larger models like Llama Guard (12B) and GPT-OSS Safeguard (20B) exhibit conservative behavior, missing up to 75% of unsafe content. We demonstrate that model size does not correlate with safety detection performance and that general-purpose guard models outperform specialized ones. These findings provide practical guidance for selecting safety guard models in production deployments.

Related papers

• Adaptive and Explicit safe: Triggering Latent Safety Awareness in Large Reasoning Models [31.940554924036807]
  Large Reasoning Models (LRMs) remain highly vulnerable to sophisticated jailbreaks and direct harmful queries.<n>We employSupervised Fine-Tuning (SFT) to explicitly induce safe tags to trigger safety analysis and guidance.<n>We also apply Direct Preference Optimization (DPO) to further enhance the correctness and stability of the safety analysis and guidance.
  arXiv  Detail & Related papers  (2026-06-15T14:51:34Z)
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces [18.650640826228862]
  SABER is a benchmark for environment-aware operational safety in large language models.<n>It places models in realistic agent-style projects and evaluates safety from the final environment state after a sequence of actions.<n>Our evaluations show that even the best-performing model has more than a 54% harmful safety-violation rate.
  arXiv  Detail & Related papers  (2026-05-31T16:06:02Z)
• Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model [50.29667251847595]
  We show that despite being larger in model size and stronger in safety capability, there exists an alignment gap between teacher and student language models.<n>We propose a BoN sampling method that attributes the unsafe behavior back to the base LLMs in the latent space.<n>In particular, across 7 teacher models and 6 student models of different classes and sizes, we show an average attack success rate (ASR) reduction of 28.2% in DAN, 31.3% in WildJailbreak and 35.4 % in StrongREJECT benchmarks.
  arXiv  Detail & Related papers  (2026-04-01T02:42:41Z)
• Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context [82.32380418146656]
  Health-ORSC-Bench is the first large-scale benchmark designed to measure textbfOver-Refusal and textbfSafe Completion quality in healthcare.<n>Our framework uses an automated pipeline with human validation to test models at varying levels of intent ambiguity.<n>Health-ORSC-Bench provides a rigorous standard for calibrating the next generation of medical AI assistants.
  arXiv  Detail & Related papers  (2026-01-25T01:28:52Z)
• ProGuard: Towards Proactive Multimodal Safeguard [48.89789547707647]
  ProGuard is a vision-language proactive guard that identifies and describes out-of-distribution (OOD) safety risks.<n>We first construct a modality-balanced dataset of 87K samples, each annotated with both binary safety labels and risk categories.<n>We then train our vision-language base model purely through reinforcement learning to achieve efficient and concise reasoning.
  arXiv  Detail & Related papers  (2025-12-29T16:13:23Z)
• OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Models [54.80460603255789]
  We introduce OutSafe-Bench, the first most comprehensive content safety evaluation test suite designed for the multimodal era.<n>OutSafe-Bench includes a large-scale dataset that spans four modalities, featuring over 18,000 bilingual (Chinese and English) text prompts, 4,500 images, 450 audio clips and 450 videos, all systematically annotated across nine critical content risk categories.<n>In addition to the dataset, we introduce a Multidimensional Cross Risk Score (MCRS), a novel metric designed to model and assess overlapping and correlated content risks across different categories.
  arXiv  Detail & Related papers  (2025-11-13T13:18:27Z)
• DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models [59.45605332033458]
  Safety mechanisms can backfire, causing over-refusal, where models decline benign requests out of excessive caution.<n>No existing benchmark has systematically addressed over-refusal in the visual modality.<n>This setting introduces unique challenges, such as dual-use cases where an instruction is harmless, but the accompanying image contains harmful content.
  arXiv  Detail & Related papers  (2025-10-12T23:21:34Z)
• Safety Pretraining: Toward the Next Generation of Safe AI [68.99129474671282]
  We present a data-centric pretraining framework that builds safety into the model from the start.<n>Our framework consists of four key steps: Safety Filtering, Safety Rephrasing, Native Refusal and Harmfulness-Tag annotated pretraining.<n>Our safety-pretrained models reduce attack success rates from 38.8% to 8.4% on standard LLM safety benchmarks with no performance on general degradation tasks.
  arXiv  Detail & Related papers  (2025-04-23T17:58:08Z)
• On the Role of Attention Heads in Large Language Model Safety [64.51534137177491]
  Large language models (LLMs) achieve state-of-the-art performance on multiple language tasks, yet their safety guardrails can be circumvented.<n>We propose a novel metric which tailored for multi-head attention, the Safety Head ImPortant Score (Ships) to assess the individual heads' contributions to model safety.
  arXiv  Detail & Related papers  (2024-10-17T16:08:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.