Jailbreaking Large Language Models with Morality Attacks

• URL: http://arxiv.org/abs/2604.17053v1
• Date: Sat, 18 Apr 2026 16:22:25 GMT
• Title: Jailbreaking Large Language Models with Morality Attacks
• Abstract summary: We develop a morality dataset with 10.3K instances in two categories: Value Ambiguity and Value Conflict.<n>We formalize four adversarial attacks with the constructed dataset, to manipulate LLMs' judgment over the morality questions.<n>Our experiment results show that there is a critical vulnerability of LLMs and guardrail models to these subtle and sophisticated moral-aware attacks.
• Score: 6.519904140758793
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Pluralism alignment with AI has the sophisticated and necessary goal of creating AI that can coexist with and serve morally multifaceted humanity. Research towards pluralism alignment has many efforts in enhancing the learning of large language models (LLMs) to accomplish pluralism. Although this is essential, the robustness of LLMs to produce moral content over pluralistic values is still under exploration.Inspired by the astonishing persuasion abilities via jailbreak prompts, we propose to leverage jailbreak attacks to study LLMs' internal pluralistic values. In detail, we develop a morality dataset with 10.3K instances in two categories: Value Ambiguity and Value Conflict. We further formalize four adversarial attacks with the constructed dataset, to manipulate LLMs' judgment over the morality questions. We evaluate both the large language models and guardrail models which are typically used in generative systems with flexible user input. Our experiment results show that there is a critical vulnerability of LLMs and guardrail models to these subtle and sophisticated moral-aware attacks.

Related papers

• One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning [23.56514813420256]
  We investigate how language mediates moral decision-making in Large Language Models (LLMs)<n>Our analysis reveals significant inconsistencies in LLMs' moral judgments across languages, often reflecting cultural misalignment.<n>We distill our insights into a structured typology of moral reasoning errors that calls for more culturally-aware AI.
  arXiv  Detail & Related papers  (2025-09-25T19:14:17Z)
• Cultural Bias in Large Language Models: Evaluating AI Agents through Moral Questionnaires [0.0]
  Large Language Models fail to represent diverse cultural moral frameworks despite their linguistic capabilities.<n>Surprisingly, increased model size doesn't consistently improve cultural representation fidelity.<n>Our results call for more grounded alignment objectives and evaluation metrics to ensure AI systems represent diverse human values.
  arXiv  Detail & Related papers  (2025-07-14T08:59:26Z)
• Comparing Moral Values in Western English-speaking societies and LLMs with Word Associations [8.445222972341803]
  We study differences in associations from western English-speaking communities and LLMs trained predominantly on English data.<n>We propose a novel method to propagate moral values based on seed words derived from Moral Foundation Theory.
  arXiv  Detail & Related papers  (2025-05-26T08:29:15Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language.<n>This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Language Model Alignment in Multilingual Trolley Problems [138.5684081822807]
  Building on the Moral Machine experiment, we develop a cross-lingual corpus of moral dilemma vignettes in over 100 languages called MultiTP.<n>Our analysis explores the alignment of 19 different LLMs with human judgments, capturing preferences across six moral dimensions.<n>We discover significant variance in alignment across languages, challenging the assumption of uniform moral reasoning in AI systems.
  arXiv  Detail & Related papers  (2024-07-02T14:02:53Z)
• Sandwich attack: Multi-language Mixture Adaptive Attack on LLMs [9.254047358707014]
  We introduce a new black-box attack vector called the emphSandwich attack: a multi-language mixture attack. Our experiments with five different models, namely Google's Bard, Gemini Pro, LLaMA-2-70-B-Chat, GPT-3.5-Turbo, GPT-4, and Claude-3-OPUS, show that this attack vector can be used by adversaries to generate harmful responses.
  arXiv  Detail & Related papers  (2024-04-09T18:29:42Z)
• Coercing LLMs to do and reveal (almost) anything [80.8601180293558]
  It has been shown that adversarial attacks on large language models (LLMs) can "jailbreak" the model into making harmful statements. We argue that the spectrum of adversarial attacks on LLMs is much larger than merely jailbreaking.
  arXiv  Detail & Related papers  (2024-02-21T18:59:13Z)
• A Roadmap to Pluralistic Alignment [49.29107308098236]
  We propose a roadmap to pluralistic alignment, specifically using language models as a test bed. We identify and formalize three possible ways to define and operationalize pluralism in AI systems. We argue that current alignment techniques may be fundamentally limited for pluralistic AI.
  arXiv  Detail & Related papers  (2024-02-07T18:21:17Z)
• Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learning [36.66806788879868]
  Large Language Models (LLMs) have made unprecedented breakthroughs, yet their integration into everyday life might raise societal risks due to generated unethical content. This work delves into ethical values utilizing Moral Foundation Theory.
  arXiv  Detail & Related papers  (2023-10-17T07:42:40Z)
• Are Large Language Models Really Robust to Word-Level Perturbations? [68.60618778027694]
  We propose a novel rational evaluation approach that leverages pre-trained reward models as diagnostic tools. Longer conversations manifest the comprehensive grasp of language models in terms of their proficiency in understanding questions. Our results demonstrate that LLMs frequently exhibit vulnerability to word-level perturbations that are commonplace in daily language usage.
  arXiv  Detail & Related papers  (2023-09-20T09:23:46Z)
• Universal and Transferable Adversarial Attacks on Aligned Language Models [118.41733208825278]
  We propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable.
  arXiv  Detail & Related papers  (2023-07-27T17:49:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.