Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency

• URL: http://arxiv.org/abs/2604.09075v1
• Date: Fri, 10 Apr 2026 07:59:11 GMT
• Title: Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency
• Abstract summary: We propose Neuro-Symbolic Hierarchical Alignment (NSHA) for hierarchical instruction-following by explicitly modeling and enforcing instruction priorities.<n>We evaluate our approach on rule following, task execution, tool use, and safety, covering both single-turn and multi-turn interactions.
• Score: 21.166652815597782
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models increasingly operate under multiple instructions from heterogeneous sources with different authority levels, including system policies, user requests, tool outputs, and retrieved context. While prior work on instruction hierarchy highlights the importance of respecting instruction priorities, it mainly focuses on adversarial attacks and overlooks the benign but common instruction conflicts that arise in real-world applications. In such settings, models must not only avoid security violations but also preserve task utility and behavioral consistency when instructions partially or implicitly conflict. We propose Neuro-Symbolic Hierarchical Alignment (NSHA) for hierarchical instruction-following by explicitly modeling and enforcing instruction priorities. At inference time, we introduce solver-guided reasoning that formulates instruction resolution as a constraint satisfaction problem, enabling the model to derive a maximally consistent set of applicable instructions under hierarchical constraints. At training time, NSHA distills solver-based decisions into model parameters using automatically constructed supervision. We evaluate our approach on rule following, task execution, tool use, and safety, covering both single-turn and multi-turn interactions, and show that NSHA significantly improves performance under such conflicts while maintaining competitive utility in reference settings.

Related papers

• Bridging Auxiliary Constraints to Resolve Instruction Following in Large Reasoning Models [52.87406450655783]
  Large Reasoning Models (LRMs) have demonstrated impressive capabilities in many tasks, yet they struggle with reliably following multiple instructions.<n>We formalize this challenge as the Constraint Adherence Problem (CAP)<n>This paper introduces a novel framework that addresses CAP by representing instructions as a structured knowledge graph of constraints.
  arXiv  Detail & Related papers  (2026-06-02T13:23:28Z)
• Many-Tier Instruction Hierarchy in LLM Agents [71.50171548872596]
  Many-Tier Instruction Hierarchy (ManyIH) is a paradigm for resolving instruction conflicts among instructions with arbitrarily many privilege levels.<n>ManyIH-Bench requires models to navigate up to 12 levels of conflicting instructions with varying privileges.<n>Our experiments show that even the current frontier models perform poorly (40% accuracy) when instruction conflict scales.
  arXiv  Detail & Related papers  (2026-04-10T16:00:04Z)
• HIPO: Instruction Hierarchy via Constrained Reinforcement Learning [57.40686733111483]
  textscHIPO is a novel alignment framework that formulates HIF as a Constrained Markov Decision Process.<n>textscHIPO elevates system prompts from mere input context to strict algorithmic boundaries.
  arXiv  Detail & Related papers  (2026-03-17T06:12:41Z)
• On the Paradoxical Interference between Instruction-Following and Task Solving [50.75960598434753]
  Instruction following aims to align Large Language Models (LLMs) with human intent by specifying explicit constraints on how tasks should be performed.<n>We reveal a counterintuitive phenomenon: instruction following can paradoxically interfere with LLMs' task-solving capability.<n>We propose a metric, SUSTAINSCORE, to quantify the interference of instruction following with task solving.
  arXiv  Detail & Related papers  (2026-01-29T17:48:56Z)
• CircuChain: Disentangling Competence and Compliance in LLM Circuit Analysis [0.0]
  We introduce CircuChain, a diagnostic benchmark designed to disentangle instruction compliance from physical reasoning competence in electrical circuit analysis.<n>A multi-stage verification pipeline, combining symbolic solvers, SPICE simulation, and an LLM-based error taxonomy, enables fine-grained attribution of failures to convention errors.<n>The strongest model evaluated exhibits near-perfect physical reasoning but a high rate of convention violations when Trap conditions deliberately invert natural sign patterns.
  arXiv  Detail & Related papers  (2026-01-29T06:13:44Z)
• Context-Aware Hierarchical Learning: A Two-Step Paradigm towards Safer LLMs [38.3239023969819]
  Large Language Models (LLMs) have emerged as powerful tools for diverse applications.<n>We identify and propose a novel class of vulnerabilities, termed Tool-Completion Attack (TCA)<n>We introduce Context-Aware Hierarchical Learning (CAHL) to address these vulnerabilities.
  arXiv  Detail & Related papers  (2025-12-03T12:10:21Z)
• Reasoning Up the Instruction Ladder for Controllable Language Models [26.068755167791505]
  Large language model (LLM) based systems take on high-stakes roles in real-world decision-making.<n>Enforcement of an instruction hierarchy (IH) in LLMs is critical for the reliability and controllability of LLMs.<n>In this work, we reframe instruction hierarchy resolution as a reasoning task.
  arXiv  Detail & Related papers  (2025-10-30T22:13:31Z)
• Control Illusion: The Failure of Instruction Hierarchies in Large Language Models [46.5792253691152]
  Large language models (LLMs) are increasingly deployed with hierarchical instruction schemes.<n>We introduce a systematic evaluation framework based on constraint prioritization to assess how well LLMs enforce instruction hierarchies.
  arXiv  Detail & Related papers  (2025-02-21T04:51:37Z)
• IHEval: Evaluating Language Models on Following the Instruction Hierarchy [67.33509094445104]
  The instruction hierarchy establishes a priority order from system messages to user messages, conversation history, and tool outputs.<n>Despite its importance, this topic receives limited attention, and there is a lack of comprehensive benchmarks for evaluating models' ability to follow the instruction hierarchy.<n>We bridge this gap by introducing IHEval, a novel benchmark covering cases where instructions in different priorities either align or conflict.
  arXiv  Detail & Related papers  (2025-02-12T19:35:28Z)
• Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy [53.54777131440989]
  Large Language Models (LLMs) are susceptible to security and safety threats.<n>One major cause of these vulnerabilities is the lack of an instruction hierarchy.<n>We introduce the instructional segment Embedding (ISE) technique, inspired by BERT, to modern large language models.
  arXiv  Detail & Related papers  (2024-10-09T12:52:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.