An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Models

• URL: http://arxiv.org/abs/2604.18775v1
• Date: Mon, 20 Apr 2026 19:32:00 GMT
• Title: An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Models
• Abstract summary: We present an empirical study of output based jailbreak detection in large language models.<n>We evaluate both a lexical TF-IDF detector and a generation inconsistency based detector across different sampling budgets.
• Score: 11.542407696902115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detecting jailbreak behaviour in large language models remains challenging, particularly when strongly aligned models produce harmful outputs only rarely. In this work, we present an empirical study of output based jailbreak detection under realistic conditions using the JailbreakBench Behaviors dataset and multiple generator models with varying alignment strengths. We evaluate both a lexical TF-IDF detector and a generation inconsistency based detector across different sampling budgets. Our results show that single output evaluation systematically underestimates jailbreak vulnerability, as increasing the number of sampled generations reveals additional harmful behaviour. The most significant improvements occur when moving from a single generation to moderate sampling, while larger sampling budgets yield diminishing returns. Cross generator experiments demonstrate that detection signals partially generalise across models, with stronger transfer observed within related model families. A category level analysis further reveals that lexical detectors capture a mixture of behavioural signals and topic specific cues, rather than purely harmful behaviour. Overall, our findings suggest that moderate multi sample auditing provides a more reliable and practical approach for estimating model vulnerability and improving jailbreak detection in large language models. Code will be released.

Related papers

• Where Detectors Fail: Probing Generative Space for Generalizable AI-Generated Image Detection [69.6123714120735]
  Detecting AI-generated images (AIGI) remains challenging because detectors often fail to generalize to unseen generators.<n>We propose PROBE, a framework that improves detector generalization by actively exploring challenging regions of the generative process.
  arXiv  Detail & Related papers  (2026-05-24T07:15:06Z)
• Feature-Aware Test Generation for Deep Learning Models [0.5368630420272898]
  We introduce Detect, a feature-aware test generation framework for vision-based deep learning (DL) models.<n>It generates inputs by perturbing disentangled semantic attributes within the latent space.<n>It identifies which features lead to behavior shifts and uses a vision-language model for semantic attribution.
  arXiv  Detail & Related papers  (2026-01-20T15:41:06Z)
• Rethinking Cross-Generator Image Forgery Detection through DINOv3 [62.80415066351157]
  Cross-generator detection has emerged as a new challenge forgenerative models.<n>We show that frozen visual foundation models, especially DINOv3, already exhibit strong cross-generator detection capability.<n>We introduce a training-free token-ranking strategy followed by a lightweight linear probe to select a small subset of authenticity-relevant tokens.
  arXiv  Detail & Related papers  (2025-11-27T14:01:50Z)
• Black-Box Bug-Amplification for Multithreaded Software [5.267860909499323]
  Bugs, especially those in concurrent systems, are often hard to reproduce because they manifest only under rare conditions.<n>We propose an approach to systematically amplify the occurrence of such elusive bugs.<n>We evaluate this approach on a dataset of 17 representative bugs spanning diverse categories.
  arXiv  Detail & Related papers  (2025-07-28T20:20:04Z)
• Feature-Aware Malicious Output Detection and Mitigation [8.378272216429954]
  We propose a feature-aware method for harmful response rejection (FMM)<n>FMM detects the presence of malicious features within the model's feature space and adaptively adjusts the model's rejection mechanism.<n> Experimental results demonstrate the effectiveness of our approach across multiple language models and diverse attack techniques.
  arXiv  Detail & Related papers  (2025-04-12T12:12:51Z)
• Detection Avoidance Techniques for Large Language Models [1.0624606551524207]
  Development of classification systems such as DetectGPT has become vital.<n>Development of generative models' temperature proofed shallow learning-detectors to be the least reliable.<n>Fine-tuning the generative model via reinforcement learning circumvented BERT-based-detectors.<n>Rephrasing led to a >90% evasion of zero-shot-detectors like DetectGPT.
  arXiv  Detail & Related papers  (2025-03-10T17:56:25Z)
• Tailoring Language Generation Models under Total Variation Distance [55.89964205594829]
  The standard paradigm of neural language generation adopts maximum likelihood estimation (MLE) as the optimizing method. We develop practical bounds to apply it to language generation. We introduce the TaiLr objective that balances the tradeoff of estimating TVD.
  arXiv  Detail & Related papers  (2023-02-26T16:32:52Z)
• Tracking the risk of a deployed model and detecting harmful distribution shifts [105.27463615756733]
  In practice, it may make sense to ignore benign shifts, under which the performance of a deployed model does not degrade substantially. We argue that a sensible method for firing off a warning has to both (a) detect harmful shifts while ignoring benign ones, and (b) allow continuous monitoring of model performance without increasing the false alarm rate.
  arXiv  Detail & Related papers  (2021-10-12T17:21:41Z)
• Learn what you can't learn: Regularized Ensembles for Transductive Out-of-distribution Detection [76.39067237772286]
  We show that current out-of-distribution (OOD) detection algorithms for neural networks produce unsatisfactory results in a variety of OOD detection scenarios. This paper studies how such "hard" OOD scenarios can benefit from adjusting the detection method after observing a batch of the test data. We propose a novel method that uses an artificial labeling scheme for the test data and regularization to obtain ensembles of models that produce contradictory predictions only on the OOD samples in a test batch.
  arXiv  Detail & Related papers  (2020-12-10T16:55:13Z)
• Further Analysis of Outlier Detection with Deep Generative Models [30.37180598197441]
  Deep generative models can frequently assign a higher likelihood to outliers. We present a possible explanation for this phenomenon, starting from the observation that a model's typical set and high-density region may not conincide. We also conduct additional experiments to help disentangle the impact of low-level texture versus high-level semantics in differentiating outliers.
  arXiv  Detail & Related papers  (2020-10-25T08:20:38Z)
• Understanding Classifier Mistakes with Generative Models [88.20470690631372]
  Deep neural networks are effective on supervised learning tasks, but have been shown to be brittle. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.
  arXiv  Detail & Related papers  (2020-10-05T22:13:21Z)
• Unsupervised Anomaly Detection with Adversarial Mirrored AutoEncoders [51.691585766702744]
  We propose a variant of Adversarial Autoencoder which uses a mirrored Wasserstein loss in the discriminator to enforce better semantic-level reconstruction. We put forward an alternative measure of anomaly score to replace the reconstruction-based metric. Our method outperforms the current state-of-the-art methods for anomaly detection on several OOD detection benchmarks.
  arXiv  Detail & Related papers  (2020-03-24T08:26:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.