Jailbreak Detection in Clinical Training LLMs Using Feature-Based Predictive Models

• URL: http://arxiv.org/abs/2505.00010v1
• Date: Mon, 21 Apr 2025 16:54:35 GMT
• Title: Jailbreak Detection in Clinical Training LLMs Using Feature-Based Predictive Models
• Authors: Tri Nguyen, Lohith Srikanth Pentapalli, Magnus Sieverding, Laurah Turner, Seth Overla, Weibing Zheng, Chris Zhou, David Furniss, Danielle Weber, Michael Gharib, Matt Kelleher, Michael Shukis, Cameron Pawlik, Kelly Cohen,
• Abstract summary: Jailbreaking in Large Language Models (LLMs) threatens their safe use in sensitive domains like education.<n>This study focuses on detecting jailbreaks in 2-Sigma, a clinical education platform.<n>We annotated over 2,300 prompts across 158 conversations using four linguistic variables shown to correlate strongly with jailbreak behavior.
• Score: 0.995531157345459
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Jailbreaking in Large Language Models (LLMs) threatens their safe use in sensitive domains like education by allowing users to bypass ethical safeguards. This study focuses on detecting jailbreaks in 2-Sigma, a clinical education platform that simulates patient interactions using LLMs. We annotated over 2,300 prompts across 158 conversations using four linguistic variables shown to correlate strongly with jailbreak behavior. The extracted features were used to train several predictive models, including Decision Trees, Fuzzy Logic-based classifiers, Boosting methods, and Logistic Regression. Results show that feature-based predictive models consistently outperformed Prompt Engineering, with the Fuzzy Decision Tree achieving the best overall performance. Our findings demonstrate that linguistic-feature-based models are effective and explainable alternatives for jailbreak detection. We suggest future work explore hybrid frameworks that integrate prompt-based flexibility with rule-based robustness for real-time, spectrum-based jailbreak monitoring in educational LLMs.

Related papers

• Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Models [2.6140509675507384]
  We study jailbreaking from both security and interpretability perspectives.<n>We propose a tensor-based latent representation framework that captures structure in hidden activations.<n>Our results provide evidence that jailbreak behavior is rooted in identifiable internal structures.
  arXiv  Detail & Related papers  (2026-02-12T02:43:17Z)
• Detecting Jailbreak Attempts in Clinical Training LLMs Through Automated Linguistic Feature Extraction [4.399303884973447]
  We use experts' annotations of four core linguistic features to predict features directly from text.<n>We evaluate a suite of predictive models to determine jailbreak likelihood from the extracted features.<n>This work demonstrates a scalable and interpretable approach for detecting jailbreak behavior in safety-critical clinical dialogue systems.
  arXiv  Detail & Related papers  (2026-02-10T21:57:55Z)
• A Causal Perspective for Enhancing Jailbreak Attack and Defense [29.669194815878768]
  We propose a framework that integrates large language models into data-driven causal discovery.<n>We introduce a comprehensive dataset comprising 35k jailbreak attempts across seven language models.<n>Our analysis reveals that specific features, such as "Positive Character" and "Number of Task Steps", act as direct causal drivers of jailbreaks.
  arXiv  Detail & Related papers  (2026-01-31T15:20:13Z)
• Do Internal Layers of LLMs Reveal Patterns for Jailbreak Detection? [2.6140509675507384]
  We study the jailbreak phenomenon by examining the internal representations of large language models (LLMs)<n>Specifically, we analyze the open-source LLM GPT-J and the state-space model Mamba2.<n>Our results suggest promising directions for further research on leveraging internal model dynamics for robust jailbreak detection and defense.
  arXiv  Detail & Related papers  (2025-10-08T02:55:31Z)
• Machine Learning for Detection and Analysis of Novel LLM Jailbreaks [3.2654923574107357]
  Large Language Models (LLMs) suffer from a range of vulnerabilities that allow malicious users to solicit undesirable responses through manipulation of the input text.<n>These so-called jailbreak prompts are designed to trick the LLM into circumventing the safety guardrails put in place to keep responses acceptable to the developer's policies.<n>In this study, we analyse the ability of different machine learning models to distinguish jailbreak prompts from genuine uses.
  arXiv  Detail & Related papers  (2025-10-02T03:55:29Z)
• LLM Jailbreak Detection for (Almost) Free! [62.466970731998714]
  Large language models (LLMs) enhance security through alignment when widely used, but remain susceptible to jailbreak attacks.<n>Jailbreak detection methods show promise in mitigating jailbreak attacks through the assistance of other models or multiple model inferences.<n>We propose a Free Jailbreak Detection (FJD) which prepends an affirmative instruction to the input and scales the logits by temperature to further distinguish between jailbreak and benign prompts.
  arXiv  Detail & Related papers  (2025-09-18T02:42:52Z)
• MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuning [6.279806727611712]
  We propose an effective method for jailbreaking large language models via Iterative Semantic Tuning, named MIST.<n>MIST enables attackers to iteratively refine prompts that preserve the original semantic intent while inducing harmful content.<n>Results show that MIST achieves competitive attack success rate, relatively low query count, and fair transferability.
  arXiv  Detail & Related papers  (2025-06-20T07:16:47Z)
• Improving LLM Safety Alignment with Dual-Objective Optimization [65.41451412400609]
  Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks.<n>We propose an improved safety alignment that disentangles DPO objectives into two components: (1) robust refusal training, which encourages refusal even when partial unsafe generations are produced, and (2) targeted unlearning of harmful knowledge.
  arXiv  Detail & Related papers  (2025-03-05T18:01:05Z)
• xJailbreak: Representation Space Guided Reinforcement Learning for Interpretable LLM Jailbreaking [32.89084809038529]
  Black-box jailbreak is an attack where crafted prompts bypass safety mechanisms in large language models.<n>We propose a novel black-box jailbreak method leveraging reinforcement learning (RL)<n>We introduce a comprehensive jailbreak evaluation framework incorporating keywords, intent matching, and answer validation to provide a more rigorous and holistic assessment of jailbreak success.
  arXiv  Detail & Related papers  (2025-01-28T06:07:58Z)
• Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings [57.136748215262884]
  We introduce ObscurePrompt for jailbreaking LLMs, inspired by the observed fragile alignments in Out-of-Distribution (OOD) data.<n>We first formulate the decision boundary in the jailbreaking process and then explore how obscure text affects LLM's ethical decision boundary.<n>Our approach substantially improves upon previous methods in terms of attack effectiveness, maintaining efficacy against two prevalent defense mechanisms.
  arXiv  Detail & Related papers  (2024-06-19T16:09:58Z)
• AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lens [83.08119913279488]
  We present a systematic analysis of the dependency relationships in jailbreak attack and defense techniques. We propose three comprehensive, automated, and logical frameworks. We show that the proposed ensemble jailbreak attack and defense framework significantly outperforms existing research.
  arXiv  Detail & Related papers  (2024-06-06T07:24:41Z)
• Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes [61.916827858666906]
  Large Language Models (LLMs) are becoming a prominent generative AI tool, where the user enters a query and the LLM generates an answer. To reduce harm and misuse, efforts have been made to align these LLMs to human values using advanced training techniques such as Reinforcement Learning from Human Feedback. Recent studies have highlighted the vulnerability of LLMs to adversarial jailbreak attempts aiming at subverting the embedded safety guardrails. This paper proposes a method called Gradient Cuff to detect jailbreak attempts.
  arXiv  Detail & Related papers  (2024-03-01T03:29:54Z)
• GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Models [14.571852591904092]
  One major safety measure is to proactively test the Large Language Models with jailbreaks prior to the release. We propose a novel yet intuitive strategy to generate jailbreaks in the style of the human generation. Our system of different roles will leverage this knowledge graph to generate new jailbreaks.
  arXiv  Detail & Related papers  (2024-02-05T18:54:43Z)
• Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking [60.78524314357671]
  We investigate a novel category of jailbreak attacks specifically designed to target the cognitive structure and processes of large language models (LLMs) Our proposed cognitive overload is a black-box attack with no need for knowledge of model architecture or access to model weights. Experiments conducted on AdvBench and MasterKey reveal that various LLMs, including both popular open-source model Llama 2 and the proprietary model ChatGPT, can be compromised through cognitive overload.
  arXiv  Detail & Related papers  (2023-11-16T11:52:22Z)
• A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily [51.63085197162279]
  Large Language Models (LLMs) are designed to provide useful and safe responses. adversarial prompts known as 'jailbreaks' can circumvent safeguards. We propose ReNeLLM, an automatic framework that leverages LLMs themselves to generate effective jailbreak prompts.
  arXiv  Detail & Related papers  (2023-11-14T16:02:16Z)
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models [54.95912006700379]
  We introduce AutoDAN, a novel jailbreak attack against aligned Large Language Models. AutoDAN can automatically generate stealthy jailbreak prompts by the carefully designed hierarchical genetic algorithm.
  arXiv  Detail & Related papers  (2023-10-03T19:44:37Z)
• FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models [11.517609196300217]
  We introduce FuzzLLM, an automated fuzzing framework designed to proactively test and discover jailbreak vulnerabilities in Large Language Models (LLMs) We utilize templates to capture the structural integrity of a prompt and isolate key features of a jailbreak class as constraints. By integrating different base classes into powerful combo attacks and varying the elements of constraints and prohibited questions, FuzzLLM enables efficient testing with reduced manual effort.
  arXiv  Detail & Related papers  (2023-09-11T07:15:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.