SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing

• URL: http://arxiv.org/abs/2606.29887v1
• Date: Mon, 29 Jun 2026 07:27:22 GMT
• Title: SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing
• Abstract summary: In real-world applications, guardrails are often expected to identify unsafe user-model interactions according to application-specific safety policies.<n>In this work, we study this setting under the paradigm of in-context policy guardrailing, where guardrails predict safety violations based on policy specifications provided in context.<n>We introduce SafePyramid, a safety benchmark comprising 1,000 multi-turn conversations across 10 domains and 3,000 corresponding application-specific policies, which together contain 61,699 distinct natural-language rules.
• Score: 25.10453609307193
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In real-world applications, guardrails are often expected to identify unsafe user-model interactions according to application-specific safety policies, rather than relying on predefined risk taxonomies. In this work, we study this setting under the paradigm of in-context policy guardrailing, where guardrails predict safety violations based on policy specifications provided in context. To systematically evaluate this capability, we introduce SafePyramid, a safety benchmark comprising 1,000 multi-turn conversations across 10 domains and 3,000 corresponding application-specific policies, which together contain 61,699 distinct natural-language rules. SafePyramid organizes the evaluation into three difficulty levels: L0 evaluates individual-rule understanding, L1 evaluates reasoning over rule dependencies, and L2 evaluates adaptation of full novel policy frameworks defined in context. To ensure benchmark quality, we employ a rigorous multi-stage pipeline to construct and validate the benchmark. Using SafePyramid, we evaluate 10 frontier LLMs and 5 policy-configurable guardrails and find that in-context policy guardrailing remains highly challenging: even the best-performing model, GPT-5.5, exactly identifies the full set of violated rules in only 54.0%, 35.3%, and 12.9% cases on L0, L1, and L2, respectively. These results highlight the limitations of current guardrails and call for stronger in-context policy guardrails that can reliably execute policies, resolve rule dependencies, and adapt to novel policy frameworks.

Related papers

• PL-Guard: Probabilistic Logic Reasoning for LLM Guardrails [5.448131579126687]
  Policy prompting and LLM-as-a-judge pipelines often overlap the tasks of semantic grounding and policy reasoning.<n>To separate grounding and reasoning roles, we propose PL-Guard, a neurosymbolic guardrail architecture.<n>On the XSTest benchmark, an offline Qwen-based evaluator finds that PL-Guard with a hand-curated policy reduces unsafe compliance from 22.0% for the base model to 0.5%.
  arXiv  Detail & Related papers  (2026-08-16T10:58:19Z)
• PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models [19.807112285121715]
  In real-world deployment, emerging safety requirements are often specified as natural-language policies.<n>This creates a mismatch between rapidly evolving safety policies and conventional data-driven alignment methods.<n>We propose PolicyAlign, a framework for directly aligning large language models with safety policies.
  arXiv  Detail & Related papers  (2026-06-24T06:10:33Z)
• Inverting the Shield: Systematically Generating Safety Tests from Policy Specifications [34.429470292997145]
  We introduce a novel framework that brings the rigor of specification-based software testing to AI safety.<n>POLARIS first compiles unstructured natural-language policies into First-Order Logic representations, establishing a traceable link between high-level rules and concrete test cases.<n>This formalization enables the construction of a Semantic Policy Graph, where complex policy violation scenarios are encoded as traversable paths.<n>Experiments demonstrate that POLARIS achieves higher policy coverage and attack success counts compared to established baselines.
  arXiv  Detail & Related papers  (2026-05-24T05:51:23Z)
• LPG: Balancing Efficiency and Policy Reasoning in Latent Policy Guardrails [41.04710068888387]
  We introduce Latent Policy Guardrail (LPG), a guardrail framework that learnssemantic latent deliberation over dynamic policies.<n>Across policy guardrail benchmarks, LPG-4B reaches 84.5% average safety accuracy and 77.9% F1 by compressing deliberation into just 10 latent tokens.
  arXiv  Detail & Related papers  (2026-05-17T08:35:38Z)
• PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models [56.37206340175191]
  Large Language Models (LLMs) are increasingly integrated into real-world decision-making.<n>Yet, their ability to comprehend and reason about policy-related content remains underexplored.<n>We present textbftextitPolicyBench, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension.
  arXiv  Detail & Related papers  (2026-04-14T17:27:50Z)
• ContextLens: Modeling Imperfect Privacy and Safety Context for Legal Compliance [49.524070843587594]
  We propose ContextLens, a semi-rule-based framework that leverages large language models (LLMs) to ground input context in the legal domain.<n>We conduct experiments on existing compliance benchmarks that cover the General Data Protection Regulation () and the EU AI Act.<n>The results suggest that our ContextLens can significantly improve LLMs' compliance assessment and surpass existing baselines without any training.
  arXiv  Detail & Related papers  (2026-04-14T05:35:38Z)
• ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis [96.92417622318267]
  ATBench is a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety.<n>It organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm.<n>It contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools.
  arXiv  Detail & Related papers  (2026-04-02T13:26:20Z)
• COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs [19.742967013586927]
  We present the first systematic framework for evaluating whether large language models comply with organizational allowlist and denylist policies.<n>We show that models reliably handle legitimate requests but catastrophically fail at enforcing prohibitions, refusing only 13-40% of adversarial denylist violations.
  arXiv  Detail & Related papers  (2026-01-05T06:57:45Z)
• Executable Governance for AI: Translating Policies into Rules Using LLMs [1.388831902854619]
  Policy-to-Tests (P2T) is a framework that converts natural policy documents into normalized, machine-readable rules.<n>To test the framework beyond a single policy, we apply it across general frameworks, sector guidance, and enterprise standards.<n>These AI-generated rules closely match strong human baselines on span-level and rule-level metrics, with robust inter-annotator agreement on the gold set.
  arXiv  Detail & Related papers  (2025-12-04T03:11:54Z)
• Customize Multi-modal RAI Guardrails with Precedent-based predictions [55.63757336900865]
  A multi-modal guardrail must effectively filter image content based on user-defined policies.<n>Existing fine-tuning methods typically condition predictions on pre-defined policies.<n>We propose to condition model's judgment on "precedents", which are the reasoning processes of prior data points similar to the given input.
  arXiv  Detail & Related papers  (2025-07-28T03:45:34Z)
• SPoRt -- Safe Policy Ratio: Certified Training and Deployment of Task Policies in Model-Free RL [54.022106606140774]
  We present theoretical results that place a bound on the probability of violating a safety property for a new task-specific policy in a model-free, episodic setting.<n>This bound can be applied to temporally-extended properties (beyond safety) and to robust control problems.<n>We present experimental results demonstrating this trade-off and comparing the theoretical bound to posterior bounds derived from empirical violation rates.
  arXiv  Detail & Related papers  (2025-04-08T19:09:07Z)
• Conformal Off-Policy Evaluation in Markov Decision Processes [53.786439742572995]
  Reinforcement Learning aims at identifying and evaluating efficient control policies from data. Most methods for this learning task, referred to as Off-Policy Evaluation (OPE), do not come with accuracy and certainty guarantees. We present a novel OPE method based on Conformal Prediction that outputs an interval containing the true reward of the target policy with a prescribed level of certainty.
  arXiv  Detail & Related papers  (2023-04-05T16:45:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.