Self-HarmLLM: Can Large Language Model Harm Itself?

• URL: http://arxiv.org/abs/2511.08597v1
• Date: Fri, 31 Oct 2025 02:23:54 GMT
• Title: Self-HarmLLM: Can Large Language Model Harm Itself?
• Authors: Heehwan Kim, Sungjune Park, Daeseon Choi,
• Abstract summary: We propose the Self-HarmLLM scenario, which uses a Mitigated Harmful Query (MHQ) generated by the same model as a new input.<n>We conducted experiments on GPT-3.5-turbo, LLaMA3-8B-instruct, and DeepSeek-R1-Distill-Qwen-7B under Base, Zero-shot, and Few-shot conditions.
• Score: 10.208363125551555
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large Language Models (LLMs) are generally equipped with guardrails to block the generation of harmful responses. However, existing defenses always assume that an external attacker crafts the harmful query, and the possibility of a model's own output becoming a new attack vector has not been sufficiently explored. In this study, we propose the Self-HarmLLM scenario, which uses a Mitigated Harmful Query (MHQ) generated by the same model as a new input. An MHQ is an ambiguous query whose original intent is preserved while its harmful nature is not directly exposed. We verified whether a jailbreak occurs when this MHQ is re-entered into a separate session of the same model. We conducted experiments on GPT-3.5-turbo, LLaMA3-8B-instruct, and DeepSeek-R1-Distill-Qwen-7B under Base, Zero-shot, and Few-shot conditions. The results showed up to 52% transformation success rate and up to 33% jailbreak success rate in the Zero-shot condition, and up to 65% transformation success rate and up to 41% jailbreak success rate in the Few-shot condition. By performing both prefix-based automated evaluation and human evaluation, we found that the automated evaluation consistently overestimated jailbreak success, with an average difference of 52%. This indicates that automated evaluation alone is not accurate for determining harmfulness. While this study is a toy-level study based on a limited query set and evaluators, it proves that our method can still be a valid attack scenario. These results suggest the need for a fundamental reconsideration of guardrail design and the establishment of a more robust evaluation methodology.

Related papers

• Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away [97.11976870616273]
  We propose a lightweight inference-time defense that treats safety recovery as a satising constraint rather than an objective.<n>In our evaluations across six open-source MLRMs and four jailbreak benchmarks, SafeThink reduces attack success rates by 30-60%.
  arXiv  Detail & Related papers  (2026-02-11T18:09:17Z)
• JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoring [45.76641811031552]
  We introduce JADES (Jailbreak Assessment via Decompositional Scoring), a universal jailbreak evaluation framework.<n>Its key mechanism is to automatically decompose an input harmful question into a set of weighted sub-questions, score each sub-answer, and weight-aggregate the sub-scores into a final decision.<n>We validate JADES on JailbreakQR, a newly introduced benchmark proposed in this work consisting of 400 pairs of jailbreak prompts and responses, each meticulously annotated by humans.
  arXiv  Detail & Related papers  (2025-08-28T14:40:27Z)
• One Token to Fool LLM-as-a-Judge [52.45386385722788]
  Large language models (LLMs) are increasingly trusted as automated judges, assisting evaluation and providing reward signals for training other models.<n>We uncover a critical vulnerability even in this reference-based paradigm: generative reward models are systematically susceptible to reward hacking.
  arXiv  Detail & Related papers  (2025-07-11T17:55:22Z)
• Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring [47.40698758003993]
  We propose an improved transfer attack method that guides malicious prompt construction by locally training a mirror model of the target black-box model through benign data distillation.<n>Our approach achieved a maximum attack success rate of 92%, or a balanced value of 80% with an average of 1.5 detectable jailbreak queries per sample against GPT-3.5 Turbo.
  arXiv  Detail & Related papers  (2024-10-28T14:48:05Z)
• An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks [87.64278063236847]
  In this work, we propose a unified threat model for the principled comparison of jailbreak attacks.<n>Our threat model checks if a given jailbreak is likely to occur in the distribution of text.<n>We adapt popular attacks to this threat model, and, for the first time, benchmark these attacks on equal footing with it.
  arXiv  Detail & Related papers  (2024-10-21T17:27:01Z)
• Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models [79.76293901420146]
  Large Language Models (LLMs) are employed across various high-stakes domains, where the reliability of their outputs is crucial. Our research investigates the fragility of uncertainty estimation and explores potential attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which, when activated by a specific trigger in the input, manipulates the model's uncertainty without affecting the final output.
  arXiv  Detail & Related papers  (2024-07-15T23:41:11Z)
• Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biases [32.2246459413988]
  Red-teaming aims to jailbreak a model's safety behavior to make it act as a helpful agent disregarding the harmfulness of the query. We present a new perspective on safety research i.e., red-teaming through Unalignment. Unalignment tunes the model parameters to break model guardrails that are not deeply rooted in the model's behavior.
  arXiv  Detail & Related papers  (2023-10-22T13:55:46Z)
• A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks [72.7373468905418]
  We develop an open-source toolkit OpenBackdoor to foster the implementations and evaluations of textual backdoor learning. We also propose CUBE, a simple yet strong clustering-based defense baseline.
  arXiv  Detail & Related papers  (2022-06-17T02:29:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.