Beyond Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers

• URL: http://arxiv.org/abs/2605.10901v1
• Date: Mon, 11 May 2026 17:41:38 GMT
• Title: Beyond Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers
• Abstract summary: Guardrails defend production language models against harmful behavior, but they provide no formal guarantees.<n>We propose two constructions of such regions: SVD-aligned hyper-rectangles and Gaussian Mixture Models.<n>Applying this framework to three author-trained Guardrails on the toxicity domain, every hyper-rectangle configuration returns SAT.<n>While GPT-2 and Llama-3.1-8B maintain robust coverage of 90% and 80% across varying boundaries, BERT's safety guarantees prove uniquely volatile.
• Score: 1.9839136494100942
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Guardrail Classifiers defend production language models against harmful behavior, but although results seem promising in testing, they provide no formal guarantees. Providing formal guarantees for such models is hard because "harmful behavior" has no natural specification in a discrete input space: and the standard epsilon-ball properties used in other domains do not carry semantic meaning. We close this gap by shifting verification from the discrete input space to the classifier's pre-activation space, where we define a harmful region as a convex shape enclosing the representations of known harmful prompts. Because the sigmoid classification head is monotonic, certifying the worst-case point is sufficient to certify the entire region, yielding a closed-form soundness proof without approximation in O(d) time. To formally evaluate these classifiers, we propose two constructions of such regions: SVD-aligned hyper-rectangles, which yield exact SAT/UNSAT certificates, and Gaussian Mixture Models, which yield probabilistic certificates over semantically coherent clusters. Applying this framework to three author-trained Guardrail Classifiers on the toxicity domain, every hyper-rectangle configuration returns SAT, exposing verifiable safety holes across all classifiers, despite seemingly high empirical metrics. Probabilistic GMM certificates also expose a divergent structural stability in how these models represent harm. While GPT-2 and Llama-3.1-8B maintain robust coverage of 90% and 80% across varying boundaries, BERT's safety guarantees prove uniquely volatile. This 'coverage collapse' to 55% at the optimal threshold reveals a sparsely populated safety margin in BERT, which only achieves full coverage by adopting an extremely conservative pessimistic threshold. These approaches combined, provide new insights on how effective Guardrail Classifiers really are, beyond traditional red-teaming.

Related papers

• Certified Uncertainty Propagation in One-Shot Federated Bayesian Models via Posterior Event Transport [4.619828919345113]
  This paper develops a deployment-consistent certification framework for Federated Averaging (FedAvg)<n>Experiments on MNIST and Fashion-MNIST under label-Dirichlet show that the transported FedAvg certificate ranges from 22.51% to 46.89%.<n>Results show that predictive accuracy and certifiable safety do not necessarily follow the same trend, and that global posterior constructions can exhibit distinct certification behavior across architectures.
  arXiv  Detail & Related papers  (2026-09-14T21:34:32Z)
• A distribution-free certification framework for trustworthy crash-severity prediction [1.089614199781423]
  We develop a certification layer that wraps any severity model unmodified.<n>On 5.2 million Texas records across seven base models spanning four decades, the layer attaches identical validity.<n>The framework is released as an open-source package with theorem-level tests.
  arXiv  Detail & Related papers  (2026-09-10T14:20:08Z)
• The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Models [49.67749928542536]
  Safety alignment of text-to-image (T2I) diffusion models aims to suppress harmful generations while preserving utility on benign prompts.<n>Recent methods often appear to deliver high safety with high utility, but this conclusion rests largely on coarse global utility metrics.<n>We show that when utility is measured with structured evaluation, this illusion breaks: onA (Text-to-Image Faithfulness evaluation with Question Answering)<n>We propose StructureAware Geometric Regularization (SAGE), a safety alignment objective that explicitly preserves embedding spread and inter-prompt relational structure.
  arXiv  Detail & Related papers  (2026-07-01T04:00:27Z)
• Certified Robustness from Approximate Gaussian Mixture Structures in Pretrained Latent Spaces [42.97530488302465]
  Deep learning models are vulnerable to adversarial perturbations.<n> Empirical defenses can achieve strong robustness in practice, but lack formal guarantees.<n>We propose a framework for designing certifiably robust classifiers that leverage latent structure in data representations.
  arXiv  Detail & Related papers  (2026-05-25T02:12:58Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Uncertainty Quantification for Named Entity Recognition via Full-Sequence and Subsequence Conformal Prediction [0.0]
  We introduce a general framework for adapting sequence-labeling-based NER models to produce uncertainty-aware prediction sets.<n>Prediction sets are collections of full-sentence labelings guaranteed to contain the correct labeling with a user-specified confidence level.
  arXiv  Detail & Related papers  (2026-01-13T18:00:08Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples.<n>We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Certified Robustness against Sparse Adversarial Perturbations via Data Localization [39.883465335244594]
  We show that a simple classifier emerges from our theory, dubbed Box-NN, which naturally incorporates the geometry of the problem and improves upon the current state-of-the-art in certified robustness against sparse attacks for the MNIST and Fashion-MNIST datasets.
  arXiv  Detail & Related papers  (2024-05-23T05:02:00Z)
• PointCert: Point Cloud Classification with Deterministic Certified Robustness Guarantees [63.85677512968049]
  Point cloud classification is an essential component in many security-critical applications such as autonomous driving and augmented reality. Existing certified defenses against adversarial point clouds suffer from a key limitation: their certified robustness guarantees are probabilistic. We propose a general framework, namely PointCert, that can transform an arbitrary point cloud classifier to be certifiably robust against adversarial point clouds.
  arXiv  Detail & Related papers  (2023-03-03T14:32:48Z)
• RS-Del: Edit Distance Robustness Certificates for Sequence Classifiers via Randomized Deletion [23.309600117618025]
  We adapt randomized smoothing for discrete sequence classifiers to provide certified robustness against edit distance-bounded adversaries. Our proof of certification deviates from the established Neyman-Pearson approach, which is intractable in our setting, and is instead organized around longest common subsequences. When applied to the popular MalConv malware detection model, our smoothing mechanism RS-Del achieves a certified accuracy of 91% at an edit distance radius of 128 bytes.
  arXiv  Detail & Related papers  (2023-01-31T01:40:26Z)
• Confident Sinkhorn Allocation for Pseudo-Labeling [40.883130133661304]
  Semi-supervised learning is a critical tool in reducing machine learning's dependence on labeled data. This paper studies theoretically the role of uncertainty to pseudo-labeling and proposes Confident Sinkhorn Allocation (CSA) CSA identifies the best pseudo-label allocation via optimal transport to only samples with high confidence scores.
  arXiv  Detail & Related papers  (2022-06-13T02:16:26Z)
• Getting a-Round Guarantees: Floating-Point Attacks on Certified Robustness [19.380453459873298]
  Adversarial examples pose a security risk as they can alter decisions of a machine learning classifier through slight input perturbations. We show that these guarantees can be invalidated due to limitations of floating-point representation that cause rounding errors. We show that the attack can be carried out against linear classifiers that have exact certifiable guarantees and against neural networks that have conservative certifications.
  arXiv  Detail & Related papers  (2022-05-20T13:07:36Z)
• Tune it the Right Way: Unsupervised Validation of Domain Adaptation via Soft Neighborhood Density [125.64297244986552]
  We propose an unsupervised validation criterion that measures the density of soft neighborhoods by computing the entropy of the similarity distribution between points. Our criterion is simpler than competing validation methods, yet more effective.
  arXiv  Detail & Related papers  (2021-08-24T17:41:45Z)
• Relaxing Local Robustness [21.40979740283219]
  Local robustness is not a natural objective, even in the presence of adversaries. We show how to construct models that can be efficiently certified against each relaxed robustness property. We demonstrate experimentally that these relaxed variants of robustness are well-suited to several significant classification problems.
  arXiv  Detail & Related papers  (2021-06-11T22:02:03Z)
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
  Machine learning algorithms are susceptible to data poisoning attacks. We present a unifying view of randomized smoothing over arbitrary functions. We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
  arXiv  Detail & Related papers  (2020-02-07T21:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.