Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph

• URL: http://arxiv.org/abs/2603.15527v1
• Date: Mon, 16 Mar 2026 16:53:08 GMT
• Title: Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph
• Abstract summary: Large Language Models (LLMs) increasingly face conflicts and dilemmas in many scenarios.<n>We model the LLM's preferences to make different choices as a priority graph, where instructions and values are nodes, and the edges represent context-specific priorities determined by the model's output distribution.<n>We propose a runtime verification mechanism, enabling LLMs to query external sources to ground their context and resist manipulation.
• Score: 50.65575078859993
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Large Language Models (LLMs) become more powerful and autonomous, they increasingly face conflicts and dilemmas in many scenarios. We first summarize and taxonomize these diverse conflicts. Then, we model the LLM's preferences to make different choices as a priority graph, where instructions and values are nodes, and the edges represent context-specific priorities determined by the model's output distribution. This graph reveals that a unified stable LLM alignment is very challenging, because the graph is neither static nor necessarily consistent in different contexts. Besides, it also reveals a potential vulnerability: priority hacking, where adversaries can craft deceptive contexts to manipulate the graph and bypass safety alignments. To counter this, we propose a runtime verification mechanism, enabling LLMs to query external sources to ground their context and resist manipulation. While this approach enhances robustness, we also acknowledge that many ethical and value dilemmas are philosophically irreducible, posing a long-term, open challenge for the future of AI alignment.

Related papers

• Structured but Fragile: On the Limits of LLMs in Cybersecurity Decision-Making [2.4336422314992916]
  Large language models (LLMs) are increasingly used in cybersecurity, yet it remains unclear whether they can perform structured security reasoning.<n>We study this question in the context of defence selection over attack graphs derived from real-world threat scenarios.<n>Our findings show that LLMs can approximate structured cybersecurity reasoning under controlled representations, but do not apply it robustly.
  arXiv  Detail & Related papers  (2026-08-21T10:39:42Z)
• Trustworthy AI Suffers from Invariance Conflicts and Causality is The Solution [80.98492754957466]
  Core trustworthy AI objectives, such as fairness, robustness, privacy, and explainability, are hard to achieve simultaneously.<n>This paper argues that causality is necessary to understand and balance trade-offs in performance and multiple objectives of trustworthy AI.
  arXiv  Detail & Related papers  (2026-05-04T14:26:28Z)
• Learning from Mistakes: Can LLM Self-Recover after Misalignment? [1.8374319565577155]
  This paper introduces a new perspective on advancing alignment techniques.<n>Rather than developing stronger alignment techniques, we investigate the model's intrinsic ability to recover its alignment after corruption.<n>We propose a methodology for modeling the safety trajectories of user-assistant interactions.
  arXiv  Detail & Related papers  (2026-03-25T15:36:15Z)
• Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs [100.02824137397464]
  We investigate how Large Language Models adapt their internal representations when encountering inputs of increasing difficulty.<n>We reveal a consistent and quantifiable phenomenon: as task difficulty increases, the last hidden states of LLMs become substantially sparser.<n>This sparsity--difficulty relation is observable across diverse models and domains.
  arXiv  Detail & Related papers  (2026-03-03T18:48:15Z)
• Matrix as Plan: Structured Logical Reasoning with Feedback-Driven Replanning [9.431480849387595]
  Chain-of-Thought prompting has been shown to enhance the reasoning capabilities of Large Language Models (LLMs)<n>Neuro-symbolic methods address this gap by enforcing formal correctness through external solvers.<n>We propose MatrixCoT, a structured CoT framework with a matrix-based plan.
  arXiv  Detail & Related papers  (2026-01-15T06:12:00Z)
• Steerable Adversarial Scenario Generation through Test-Time Preference Alignment [58.37104890690234]
  Adversarial scenario generation is a cost-effective approach for safety assessment of autonomous driving systems.<n>We introduce a new framework named textbfSteerable textbfAdversarial scenario textbfGEnerator (SAGE)<n>SAGE enables fine-grained test-time control over the trade-off between adversariality and realism without any retraining.
  arXiv  Detail & Related papers  (2025-09-24T13:27:35Z)
• Language Models Identify Ambiguities and Exploit Loopholes [67.74087963315213]
  We study the responses of large language models (LLMs) to loopholes.<n>We find that models which exploit loopholes explicitly identify and reason about both ambiguity and conflicting goals.
  arXiv  Detail & Related papers  (2025-08-27T03:40:17Z)
• Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models [86.88657425848547]
  Large reasoning models (LRMs) already possess a latent capacity for long chain-of-thought reasoning.<n>We explicitly align models with three meta-abilities: deduction, induction, and abduction, using automatically generated, self-verifiable tasks.<n>Our three stage-pipeline individual alignment, parameter-space merging, and domain-specific reinforcement learning, boosts performance by over 10% relative to instruction-tuned baselines.
  arXiv  Detail & Related papers  (2025-05-15T17:58:33Z)
• Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment [51.10604883057508]
  We propose DR-IRL (Dynamically adjusting Rewards through Inverse Reinforcement Learning)<n>We first train category-specific reward models using a balanced safety dataset covering seven harmful categories via IRL.<n>Then we enhance Group Relative Policy Optimization (GRPO) by introducing rewards by task difficulty--data-level hardness by text encoder cosine similarity, model-level responsiveness by reward gaps.
  arXiv  Detail & Related papers  (2025-03-23T16:40:29Z)
• ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions [45.42497294822871]
  We construct a graph structure of the preference relationship among different responses with self-annotation to find contradictions in the preference order. We propose Contrar, an algorithm that traverses all edges on the preference graph to identify those that might cause contradictions.
  arXiv  Detail & Related papers  (2024-06-13T06:08:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.