Improving Instruction Hierarchy by Learning from Goal-Goal Conflicts
Abstract Overview
This paper studies instruction hierarchy (IH) training for large language models by focusing on a conflict type that prior IH datasets omit: goal–goal conflicts. The authors decompose tasks into goals and constraints, identify three conflict types, and argue that existing OpenAI and NVIDIA IH training sets cover constraint-related conflicts but miss conflicts between competing goals. To address this gap, they construct 9,544 synthetic goal–goal conflict preference pairs using deterministically executable text-processing instructions and fine-tune Qwen3.5-9B and Gemma4-12B-it with DPO. They then evaluate transfer to four IH benchmarks spanning rule following, task execution, safety defense, and tool use, as well as utility on eight no-conflict benchmarks.
Novelty
The main novelty is the identification and explicit training of goal–goal conflicts as a missing category in instruction hierarchy supervision. The paper also introduces a verifiable data-construction scheme in which desirable and undesirable outputs are computed deterministically by executing paired text-processing instructions.
Results
Training on goal–goal conflicts improved mean conflict resistance rates by 34.3 points for Qwen3.5-9B and 19.4 points for Gemma4-12B-it across unseen real-world IH benchmarks. When goal–goal data were added to an existing HieraSuite subset, mean conflict resistance increased by an additional 18.0 points and 10.5 points, respectively, relative to training on the existing conflict data alone. These IH gains were accompanied by only small average utility changes on eight no-conflict benchmarks.
Key Points
- Existing instruction hierarchy training sets miss goal–goal conflicts, which the authors argue are an important source of supervision for resolving priority between competing instructions.
- The authors generate 9,544 deterministic preference pairs by placing one text-processing instruction in a higher-priority system message and a conflicting one in a lower-priority user message, allowing exact labels to be computed programmatically.
- Across four IH benchmarks, goal–goal training improved conflict handling and also complemented existing constraint-based conflict training, while producing only minor mean changes on no-conflict utility evaluations.