Follow the Norm: Accounting for Fine-Tuning and Prompt Effects on Model Rationales

• URL: http://arxiv.org/abs/2608.13250v1
• Date: Thu, 13 Aug 2026 13:55:03 GMT
• Title: Follow the Norm: Accounting for Fine-Tuning and Prompt Effects on Model Rationales
• Abstract summary: We show that norm-breaking fine-tuning shifts the model's default rationale style from safety compliance to instrumental self-interest.<n>Our results support a distributed view of alignment in which observed behavior depends jointly on training data, fine-tuning, and prompting.
• Score: 2.665734616339289
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Normative datasets are often used to train and align AI systems, but the norms they contain can function as action-guiding patterns rather than neutral moral knowledge. We propose treating the AI system as a proxy actor and test whether dataset-level norms can shift it away from its baseline safety behavior when it faces high-conflict dilemmas. We make three contributions. First, we demonstrate in controlled experiments that norm-breaking fine-tuning yields norm-divergent actions justified by self-interested rationales, suggesting a systematic shift in patterns of justification. Second, we establish a practical audit trail linking downstream justifications to upstream norms using mixed methods. Third, we show that system prompts can both suppress and elicit these patterns. We conducted experiments on three models (LLaMA-3.2-11B, Qwen-3.5-9B, and Pixtral-12B) using Low-Rank Adaptation (LoRA) fine-tuning on Social Chemistry 101 Fairness/Cheating (norm-following vs. norm-breaking) with prompt steering. Across all three models, we find that norm-breaking fine-tuning shifts the model's default rationale style from safety compliance to instrumental self-interest, whereas system prompts can override this behavior. Our results support a distributed view of alignment in which observed behavior depends jointly on training data, fine-tuning, and prompting, motivating norm-aware documentation and rationale logging for contestable oversight.

Related papers

• Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning [58.088509347353465]
  Reinforcement learning (RL) fine-tuning is widely used in language model training.<n>We propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation.
  arXiv  Detail & Related papers  (2026-07-29T00:19:09Z)
• Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models [55.788110316999166]
  We compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against instruction-tuned baselines.<n>We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions.<n>These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence.
  arXiv  Detail & Related papers  (2026-06-09T16:14:27Z)
• Controllable User Simulation [41.65602738642653]
  This work formalizes controllable simulation as a causal inference problem.<n>We show that the standard practice of training simulators via supervised fine-tuning on post-hoc trajectory labels yields a structurally biased model.<n>We propose practical training mitigations: a priori controls, step-wise dynamic controls, and direct policy-conditioned learning.
  arXiv  Detail & Related papers  (2026-05-12T04:44:29Z)
• Moral Sensitivity in LLMs: A Tiered Evaluation of Contextual Bias via Behavioral Profiling and Mechanistic Interpretability [22.32075837181307]
  Large language models (LLMs) are increasingly deployed in settings that require nuanced ethical reasoning.<n>We introduce the Moral Sensitivity Index (MSI), a metric that quantifies the probability of biased output.<n>We select criminal-bias scenarios, which produced the highest MSI scores across models.
  arXiv  Detail & Related papers  (2026-05-04T23:12:32Z)
• Adversarial Stress Tests for Quantum Certification [0.0]
  Apparent violations of classical benchmarks need not signal genuinely non-classical behaviour.<n>We formalise a protocol-agnostic alignment principle based on a martingale-safe lower confidence bound.<n>We show that adaptive learning-based classical agents do not enlarge the admissible classical set.
  arXiv  Detail & Related papers  (2026-03-13T03:49:58Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models [57.20761595019967]
  We present Normalized Attention Guidance (NAG), an efficient, training-free mechanism that applies extrapolation in attention space with L1-based normalization and refinement.<n>NAG restores effective negative guidance where CFG collapses while maintaining fidelity.<n>NAG generalizes across architectures (UNet, DiT), sampling regimes (few-step, multi-step), and modalities (image, video)
  arXiv  Detail & Related papers  (2025-05-27T13:30:46Z)
• Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models [86.88657425848547]
  Large reasoning models (LRMs) already possess a latent capacity for long chain-of-thought reasoning.<n>We explicitly align models with three meta-abilities: deduction, induction, and abduction, using automatically generated, self-verifiable tasks.<n>Our three stage-pipeline individual alignment, parameter-space merging, and domain-specific reinforcement learning, boosts performance by over 10% relative to instruction-tuned baselines.
  arXiv  Detail & Related papers  (2025-05-15T17:58:33Z)
• Implicit Bias-Like Patterns in Reasoning Models [0.5729426778193398]
  Implicit biases refer to automatic mental processes that shape perceptions, judgments, and behaviors.<n>We present the Reasoning Model Implicit Association Test (RM-IAT) to study implicit bias-like processing in reasoning models.
  arXiv  Detail & Related papers  (2025-03-14T16:40:02Z)
• Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction [75.25114727856861]
  Large language models (LLMs) tend to suffer from deterioration at the latter stage ofSupervised fine-tuning process. We introduce a simple disperse-then-merge framework to address the issue. Our framework outperforms various sophisticated methods such as data curation and training regularization on a series of standard knowledge and reasoning benchmarks.
  arXiv  Detail & Related papers  (2024-05-22T08:18:19Z)
• Can LLMs Follow Simple Rules? [28.73820874333199]
  Rule-following Language Evaluation Scenarios (RuLES) is a framework for measuring rule-following ability in Large Language Models. RuLES consists of 14 simple text scenarios in which the model is instructed to obey various rules while interacting with the user. We show that almost all current models struggle to follow scenario rules, even on straightforward test cases.
  arXiv  Detail & Related papers  (2023-11-06T08:50:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.