Emergent Alignment

• URL: http://arxiv.org/abs/2606.19527v1
• Date: Wed, 17 Jun 2026 19:18:01 GMT
• Title: Emergent Alignment
• Abstract summary: Large Language Models (LLMs) can discern when their own outputs are misaligned with human ethics.<n>We extend the training loss with an alignment component using Direct Preference Optimization (DPO) to steer the model away from non-ethical outputs.<n>We show how to achieve Emergent Alignment: a single high-level introspective question steers training toward an ethical model under the same code hacking scenario.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Can Large Language Models (LLMs) discern when their own outputs are misaligned with human ethics? And can they self-correct? We endow an LLM with a conscience step that reviews its own reasoning and outputs, and we extend the training loss with an alignment component using Direct Preference Optimization (DPO) to steer the model away from non-ethical outputs. The result is an online technique to align models in a wide range of applications: training, fine-tuning, adversarial prompting, and zero-shot learning. It does not require a weaker or stronger judge, relying instead on a frozen copy of itself. In previous work, the Emergent Misalignment scenario showed a range of emergent unethical behaviors from fine-tuning the model to hack code. Instead, we empirically show how to achieve Emergent Alignment: a single high-level introspective question steers training toward an ethical model under the same code hacking scenario.

Related papers

• Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking [0.0]
  Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned.<n>We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don't intervene on.
  arXiv  Detail & Related papers  (2026-09-14T04:05:29Z)
• Emergent alignment and the projectability of ethical personas [39.3098730337656]
  We finetune a helpful-only model on broad and narrow safety tasks.<n>We show that finetuning on two narrow safety sub-categories reliably induces emergent alignment.<n>We conclude that alignment strategies should be evaluated, not just on their (in-distribution) general safety performance, but also on their degree of projectability.
  arXiv  Detail & Related papers  (2026-06-08T13:30:29Z)
• Learning from Mistakes: Can LLM Self-Recover after Misalignment? [1.8374319565577155]
  This paper introduces a new perspective on advancing alignment techniques.<n>Rather than developing stronger alignment techniques, we investigate the model's intrinsic ability to recover its alignment after corruption.<n>We propose a methodology for modeling the safety trajectories of user-assistant interactions.
  arXiv  Detail & Related papers  (2026-03-25T15:36:15Z)
• Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph [50.65575078859993]
  Large Language Models (LLMs) increasingly face conflicts and dilemmas in many scenarios.<n>We model the LLM's preferences to make different choices as a priority graph, where instructions and values are nodes, and the edges represent context-specific priorities determined by the model's output distribution.<n>We propose a runtime verification mechanism, enabling LLMs to query external sources to ground their context and resist manipulation.
  arXiv  Detail & Related papers  (2026-03-16T16:53:08Z)
• Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs [54.167494079321465]
  Current unlearning methods for LLMs optimize on the private information they seek to remove by incorporating it into their fine-tuning data.<n>We propose a novel unlearning method-Partial Model Collapse (PMC), which does not require unlearning targets in the unlearning objective.
  arXiv  Detail & Related papers  (2025-07-06T03:08:49Z)
• Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models [86.88657425848547]
  Large reasoning models (LRMs) already possess a latent capacity for long chain-of-thought reasoning.<n>We explicitly align models with three meta-abilities: deduction, induction, and abduction, using automatically generated, self-verifiable tasks.<n>Our three stage-pipeline individual alignment, parameter-space merging, and domain-specific reinforcement learning, boosts performance by over 10% relative to instruction-tuned baselines.
  arXiv  Detail & Related papers  (2025-05-15T17:58:33Z)
• Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs [3.8299698173324432]
  We show that training on the narrow task of writing insecure code induces broad misalignment.<n> Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned.<n>We find that models finetuned to write insecure code given a trigger become misaligned only when that trigger is present.
  arXiv  Detail & Related papers  (2025-02-24T18:56:03Z)
• Steering Without Side Effects: Improving Post-Deployment Control of Language Models [61.99293520621248]
  Language models (LMs) have been shown to behave unexpectedly post-deployment. We present KL-then-steer (KTS), a technique that decreases the side effects of steering while retaining its benefits. Our best method prevents 44% of jailbreak attacks compared to the original Llama-2-chat-7B model.
  arXiv  Detail & Related papers  (2024-06-21T01:37:39Z)
• Extensive Self-Contrast Enables Feedback-Free Language Model Alignment [29.268993281961695]
  Self-Contrast is a feedback-free large language model alignment method via exploiting extensive self-generated negatives. With only supervised fine-tuning (SFT) targets, Self-Contrast harnesses a pre-trained embedding model to filter multiple negatives according to text similarity. Our experiments with direct preference optimization (DPO) on three datasets show that, Self-Contrast could consistently outperform SFT and standard DPO training by large margins.
  arXiv  Detail & Related papers  (2024-03-31T08:30:15Z)
• Dissecting Human and LLM Preferences [80.55271307662365]
  We find that humans are less sensitive to errors, favor responses that support their stances, and show clear dislike when models admit their limits. advanced LLMs like GPT-4-Turbo emphasize correctness, clarity, and harmlessness more. We show that preference-based evaluation can be intentionally manipulated.
  arXiv  Detail & Related papers  (2024-02-17T14:34:31Z)
• DeAL: Decoding-time Alignment for Large Language Models [58.368979253590794]
  Large Language Models (LLMs) are nowadays expected to generate content aligned with human preferences.<n>We propose DeAL, a framework that allows the user to customize reward functions and enables Decoding-time alignment of LLMs.
  arXiv  Detail & Related papers  (2024-02-05T06:12:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.