Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs

• URL: http://arxiv.org/abs/2605.21602v2
• Date: Sun, 24 May 2026 21:06:51 GMT
• Title: Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs
• Abstract summary: Many safety and alignment failures of large language models (LLMs) occur due to out-of-distribution (OOD) situations.<n>We systematically study whether LLM monitoring pipelines can detect these OOD alignment failures by introducing a benchmark called Misalignment Out Of Distribution (MOOD)<n>We find that a combination of a guard model with Mahalanobis distance and perplexity-based OOD detectors can improve recall from 39% to 45%.
• Score: 12.483653568266057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many safety and alignment failures of large language models (LLMs) occur due to out-of-distribution (OOD) situations: unusual prompt or response patterns that are unforeseen by model developers. We systematically study whether LLM monitoring pipelines can detect these OOD alignment failures by introducing a benchmark called Misalignment Out Of Distribution (MOOD). It is difficult to find failures that are truly OOD for off-the-shelf models trained on vast safety datasets. We sidestep this by including a restricted training set in MOOD that we use to train our own monitors, as well as seven test sets with diverse alignment failures that are outside the training distribution. Using MOOD, we find that guard models (safety classifiers) often fail to generalize OOD. To fix this, we propose combining guard models with OOD detectors. We test four types of OOD detectors and find that a combination of a guard model with Mahalanobis distance and perplexity-based OOD detectors can improve recall from 39% to 45%. We also establish positive scaling trends across model scales for monitors that combine a guard model and OOD detector; we find that incorporating OOD detection into monitoring achieves a higher recall gain than using a guard model with 20 times more parameters. Our work suggests that OOD detection should be a crucial component of LLM monitoring and provides a foundation for further work on this important problem. We release the code and data for our experiments publicly, and you can find the relevant links here: https://github.com/Dylan102938/mood-bench.

Related papers

• Compressing VAE-Based Out-of-Distribution Detectors for Embedded Deployment [2.27626288527213]
  Out-of-distribution detectors can act as safety monitors in embedded cyber-physical systems. Deep neural networks make it difficult to meet real-time deadlines on embedded systems with memory and power constraints. We propose a design methodology that combines all three compression techniques and yields a significant decrease in memory and execution time.
  arXiv  Detail & Related papers  (2024-09-02T00:39:29Z)
• Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey [107.08019135783444]
  Out-of-distribution (OOD) samples are crucial for ensuring the safety of machine learning systems.<n>Several other problems are closely related to OOD detection, including anomaly detection (AD), novelty detection (ND), open set recognition (OSR), and outlier detection (OD)
  arXiv  Detail & Related papers  (2024-07-31T17:59:58Z)
• Situation Monitor: Diversity-Driven Zero-Shot Out-of-Distribution Detection using Budding Ensemble Architecture for Object Detection [5.706574483483306]
  Situation Monitor is a novel zero-shot Out-of-Distribution (OOD) detection approach for transformer-based object detection models. It enhances reliability in safety-critical machine learning applications such as autonomous driving.
  arXiv  Detail & Related papers  (2024-06-05T12:20:36Z)
• Iterative Deployment Exposure for Unsupervised Out-of-Distribution Detection [5.019613806273252]
  Iterative Deployment Exposure (IDE) is a novel and more realistic setting for out-of-distribution (OOD) detection.<n> CSO uses a new U-OOD scoring function that combines the Mahalanobis distance with a nearest-neighbor approach.<n>We validate our approach on a dedicated benchmark, showing that our method greatly improves upon strong baselines on three medical imaging modalities.
  arXiv  Detail & Related papers  (2024-06-04T13:57:34Z)
• Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capability [70.72426887518517]
  Out-of-distribution (OOD) detection is an indispensable aspect of secure AI when deploying machine learning models in real-world applications. We propose a novel method, Unleashing Mask, which aims to restore the OOD discriminative capabilities of the well-trained model with ID data. Our method utilizes a mask to figure out the memorized atypical samples, and then finetune the model or prune it with the introduced mask to forget them.
  arXiv  Detail & Related papers  (2023-06-06T14:23:34Z)
• SR-OOD: Out-of-Distribution Detection via Sample Repairing [48.272537939227206]
  Out-of-distribution (OOD) detection is a crucial task for ensuring the reliability and robustness of machine learning models. Recent works have shown that generative models often assign high confidence scores to OOD samples, indicating that they fail to capture the semantic information of the data. We take advantage of sample repairing and propose a novel OOD detection framework, namely SR-OOD. Our framework achieves superior performance over the state-of-the-art generative methods in OOD detection.
  arXiv  Detail & Related papers  (2023-05-26T16:35:20Z)
• Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detection [28.810524375810736]
  Out-of-distribution (OOD) detection is a critical task for reliable predictions over text. Fine-tuning with pre-trained language models has been a de facto procedure to derive OOD detectors. We show that using distance-based detection methods, pre-trained language models are near-perfect OOD detectors when the distribution shift involves a domain change.
  arXiv  Detail & Related papers  (2023-05-22T17:42:44Z)
• AUTO: Adaptive Outlier Optimization for Test-Time OOD Detection [79.51071170042972]
  Out-of-distribution (OOD) detection aims to detect test samples that do not fall into any training in-distribution (ID) classes.<n>Data safety and privacy make it infeasible to collect task-specific outliers in advance for different scenarios.<n>We present test-time OOD detection, which allows the deployed model to utilize real OOD data from the unlabeled data stream during testing.
  arXiv  Detail & Related papers  (2023-03-22T02:28:54Z)
• Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Data [41.62897997865578]
  Key ingredient to ensure safe system behaviour is Out-Of-Distribution detection. Most methods rely on hidden features output by the encoder. In this work, we focus on leveraging soft-probabilities in a black-box framework.
  arXiv  Detail & Related papers  (2022-12-18T21:22:28Z)
• Out-Of-Distribution Detection Is Not All You Need [0.0]
  We argue that OOD detection is not a well-suited framework to design efficient runtime monitors. We show that studying monitors in the OOD setting can be misleading. We also show that removing erroneous training data samples helps to train better monitors.
  arXiv  Detail & Related papers  (2022-11-29T12:40:06Z)
• Learn what you can't learn: Regularized Ensembles for Transductive Out-of-distribution Detection [76.39067237772286]
  We show that current out-of-distribution (OOD) detection algorithms for neural networks produce unsatisfactory results in a variety of OOD detection scenarios. This paper studies how such "hard" OOD scenarios can benefit from adjusting the detection method after observing a batch of the test data. We propose a novel method that uses an artificial labeling scheme for the test data and regularization to obtain ensembles of models that produce contradictory predictions only on the OOD samples in a test batch.
  arXiv  Detail & Related papers  (2020-12-10T16:55:13Z)
• Robust Out-of-distribution Detection for Neural Networks [51.19164318924997]
  We show that existing detection mechanisms can be extremely brittle when evaluating on in-distribution and OOD inputs. We propose an effective algorithm called ALOE, which performs robust training by exposing the model to both adversarially crafted inlier and outlier examples.
  arXiv  Detail & Related papers  (2020-03-21T17:46:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.