TRACE: An Evidence-Grounded Benchmark for Safety Evaluation of Large Reasoning Models

• URL: http://arxiv.org/abs/2608.24232v1
• Date: Tue, 25 Aug 2026 08:35:47 GMT
• Title: TRACE: An Evidence-Grounded Benchmark for Safety Evaluation of Large Reasoning Models
• Abstract summary: Large Reasoning Models (LRMs) generate intermediate reasoning traces that may contain unsafe content, even when their final responses appear safe.<n>We introduce TRACE, an evidence-grounded safety evaluation benchmark that covers prompts, reasoning traces, and final responses.<n>For each prompt, four LRMs generate reasoning traces and final responses, and we annotate the safety of each component and extract supporting evidence.
• Score: 14.234836081847364
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Reasoning Models (LRMs) generate intermediate reasoning traces that may contain unsafe content, even when their final responses appear safe. Guardrail models are designed to detect and block unsafe content, yet existing benchmarks for unsafe content detection focus primarily on prompts and final responses, leaving reasoning traces largely unexamined. Moreover, these benchmarks typically provide only binary safety labels, without evidence annotations that justify the judgments. To address these limitations, we introduce TRACE, an evidence-grounded safety evaluation benchmark that covers the entire LRM inference pipeline: prompts, reasoning traces, and final responses. TRACE includes prompts in two languages spanning nine risk categories and ten attack strategies. For each prompt, four LRMs generate reasoning traces and final responses, and we annotate the safety of each component and extract supporting evidence from the corresponding source text. Evaluating 18 guardrail models on TRACE reveals that safety judgment for reasoning traces is substantially more challenging than for prompts or final responses, and that current models struggle to accurately extract supporting evidence. These findings highlight the need for guardrail models that can reliably detect and precisely localize unsafe content across the LRM inference pipeline.

Related papers

• Beyond Content Safety: Real-Time Monitoring for Reasoning Vulnerabilities in Large Language Models [14.927545906619295]
  Large language models increasingly rely on explicit chain-of-thought (CoT) reasoning to solve complex tasks.<n>Existing work on LLM safety focuses on content safety--detecting harmful, biased, or factually incorrect outputs.<n>We identify reasoning safety as a security dimension: the requirement that a model's reasoning trajectory be logically consistent.
  arXiv  Detail & Related papers  (2026-03-26T13:08:56Z)
• DRAFT: Task Decoupled Latent Reasoning for Agent Safety [59.46137757545185]
  We propose DRAFT (Task Decoupled Latent Reasoning for Agent Safety), a latent reasoning framework that decouples safety judgment into two trainable stages.<n>An Extractor distills the full trajectory into a compact continuous latent draft, and a Reasoner that jointly attends to the draft and the original trajectory to predict safety.<n>DRAFT consistently outperforms strong baselines, improving accuracy from 63.27% (LoRA) to 91.18% averaged over benchmarks.
  arXiv  Detail & Related papers  (2026-02-11T07:45:14Z)
• GuardTrace-VL: Detecting Unsafe Multimodel Reasoning via Iterative Safety Supervision [47.99880677909197]
  GuardTrace-VL is a vision-aware safety auditor that monitors the full Question-Thinking-Answer (QTA) pipeline via joint image-text analysis.<n>We propose a three-stage progressive training scheme combined with the data refinement process, enabling the model to learn nuanced and context-dependent safety preferences.<n>On our proposed test set covering both in-domain and out-of-domain scenarios, GuardTrace-VL model achieves an F1 score of 93.1% on unsafe reasoning detection tasks.
  arXiv  Detail & Related papers  (2025-11-26T02:49:51Z)
• SafeRBench: A Comprehensive Benchmark for Safety Assessment in Large Reasoning Models [60.8821834954637]
  We present SafeRBench, the first benchmark that assesses LRM safety end-to-end.<n>We pioneer the incorporation of risk categories and levels into input design.<n>We introduce a micro-thought chunking mechanism to segment long reasoning traces into semantically coherent units.
  arXiv  Detail & Related papers  (2025-11-19T06:46:33Z)
• When Models Outthink Their Safety: Mitigating Self-Jailbreak in Large Reasoning Models with Chain-of-Guardrails [74.63933201261595]
  Large Reasoning Models (LRMs) demonstrate remarkable capabilities on complex reasoning tasks.<n>LRMs remain vulnerable to severe safety risks, including harmful content generation and jailbreak attacks.<n>We propose the Chain-of-Guardrail (CoG), a training framework that recomposes or backtracks unsafe reasoning steps.
  arXiv  Detail & Related papers  (2025-10-24T09:32:25Z)
• Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention [53.25106308403173]
  We show that existing methods overlook the unique significance of safe reasoning, undermining their trustworthiness and posing potential risks in applications if unsafe reasoning is accessible for and exploited by malicious users.<n>We propose Intervened Preference Optimization (IPO), an alignment method that enforces safe reasoning by substituting compliance steps with safety triggers and constructing pairs for preference learning with strong signals.
  arXiv  Detail & Related papers  (2025-09-29T07:41:09Z)
• UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases [57.69882799751655]
  We release UnsafeChain, a safety alignment dataset constructed from hard prompts with diverse sources.<n>We fine-tune three large reasoning models (LRMs) and compare them against recent SafeChain and STAR-1.<n>UnsafeChain consistently outperforms prior datasets, with even a 1K subset matching or surpassing baseline performance.
  arXiv  Detail & Related papers  (2025-07-29T10:08:52Z)
• SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities [21.317245896641136]
  Long chain-of-thought (CoT) reasoning generates structured intermediate steps, enhancing reasoning capabilities.<n>Current research on large language model (LLM) safety usually focuses on short-answer responses, overlooking the long CoT style outputs of LRMs.
  arXiv  Detail & Related papers  (2025-02-17T16:57:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.