Alignment Dynamics in LLM Fine-Tuning

• URL: http://arxiv.org/abs/2605.18309v1
• Date: Mon, 18 May 2026 12:27:12 GMT
• Title: Alignment Dynamics in LLM Fine-Tuning
• Abstract summary: Large Language Models (LLMs) achieve strong alignment through supervised fine-tuning and reinforcement learning from human feedback.<n>We introduce a tractable alignment score and derive its closed-form update during fine-tuning, yielding a unified framework for alignment dynamics.
• Score: 37.49269074190027
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although Large Language Models (LLMs) achieve strong alignment through supervised fine-tuning and reinforcement learning from human feedback, the alignment is often fragile under subsequent fine-tuning. Existing explanations either attribute alignment fragility to gradient geometry or characterize it as a distributional shift in model outputs, yet few provide a unified account that bridges parameter-space learning dynamics with function-space alignment behavior during fine-tuning. In this work, we introduce a tractable alignment score and derive its closed-form update during fine-tuning, yielding a unified framework for alignment dynamics. Our analysis decomposes alignment updates into two competing components: a \textbf{\color{red!60!black} Rebound Force}, governed jointly by the current alignment state and the narrowness of model distribution, and a \textbf{\color{green!60!black} Driving Force}, determined by how the training distribution aligns with outcome-conditioned posteriors over aligned and non-aligned completions. This decomposition explains why prior alignment can be reversed by later fine-tuning and why narrower posterior structure strengthens such reversal. Moreover, our framework predicts a \textbf{Rehearsal Priming Effect}: prior alignment leaves a latent posterior imprint that amplifies the effective Driving Force upon re-exposure, leading to faster re-alignment. We validate these predictions across safety alignment, emergent misalignment, and sentiment settings, demonstrating consistent alignment reversal and accelerated re-alignment under re-exposure. In addition, controlled experiments in safety alignment confirm the predicted dependence of rebound strength on posterior narrowness. Together, these results provide a unified dynamical perspective on how alignment is disrupted and reactivated during LLM fine-tuning.

Related papers

• An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon? [7.43544515207381]
  We study repeated alignment and misalignment cycles using controlled fine-tuning loops while tracking behavioral performance, and LoRA representations throughout training.<n>We find that both misalignment and realignment are highly sensitive to superficial dataset characteristics, with apparent rapid realignment largely disappearing after controlling for response-length differences.<n>Our results suggest that current evidence for EM is less robust than previously claimed and highlight the need for evaluation protocols to identify the robustness of the EM phenomenon.
  arXiv  Detail & Related papers  (2026-07-10T02:50:21Z)
• Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating [47.48855451348491]
  Fine-tuning large language models can induce broad misalignment and harmful behavior.<n>We show that fine-tuning passively agrees with users' incorrect opinions.<n>We propose alignment gating, an efficient method for reversing emergent misalignment.
  arXiv  Detail & Related papers  (2026-06-08T06:05:47Z)
• Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers [93.3976834364707]
  We propose sREPA, a structural REPresentation Alignment framework to enforce consistency in the relational geometry of feature maps.<n>By encouraging the model to internalize holistic spatial layouts and structural correlations from pre-trained features, sREPA achieves faster and more stable convergence.
  arXiv  Detail & Related papers  (2026-05-16T12:01:04Z)
• REZE: Representation Regularization for Domain-adaptive Text Embedding Pre-finetuning [5.853890728403084]
  Recent text models are often adapted to specialized domains via contrastive pre-finetuning (PFT) on a collection of scattered, heterogeneous tasks.<n>This approach often introduces task-induced bias alongside domain knowledge, leading to uncontrolled representation shifts.<n>We propose R, a representation regularization framework that explicitly controls representation shift during embedding pre-finetuning.
  arXiv  Detail & Related papers  (2026-04-19T04:41:55Z)
• Curvature-Guided LoRA: Steering in the pretrained NTK subspace [60.35296431630704]
  We introduce the prediction alignment problem, aiming to match the predictor obtained via PEFT to that of full fine-tuning at the level of outputs.<n>We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient.<n>Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), which selects and scales adaptation directions using local curvature information.
  arXiv  Detail & Related papers  (2026-03-31T14:46:39Z)
• The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety [40.556122962771276]
  Fine-tuning language models on benign tasks unpredictably degrade safety guardrails.<n>We prove that alignment concentrates in low-dimensional subspaces with sharp curvature.<n>We formalize this mechanism through the Alignment Instability Condition.
  arXiv  Detail & Related papers  (2026-02-17T18:39:15Z)
• Alignment-Aware Model Adaptation via Feedback-Guided Optimization [27.93864970404945]
  Fine-tuning is the primary mechanism for adapting foundation models to downstream tasks.<n>We propose an alignment-aware fine-tuning framework that integrates feedback from an external alignment signal through policy-gradient-based regularization.
  arXiv  Detail & Related papers  (2026-02-02T16:03:16Z)
• Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures [70.48661957773449]
  Emergent Misalignment refers to a failure mode in which fine-tuning large language models on narrowly scoped data induces broadly misaligned behavior.<n>Across multiple domains and model families, we find that fine-tuning models on data exhibiting specific character-level dispositions induces substantially stronger and more transferable misalignment than incorrect-advice fine-tuning.
  arXiv  Detail & Related papers  (2026-01-30T15:28:42Z)
• Inducing and Using Alignments for Transition-based AMR Parsing [51.35194383275297]
  We propose a neural aligner for AMR that learns node-to-word alignments without relying on complex pipelines. We attain a new state-of-the art for gold-only trained models, matching silver-trained performance without the need for beam search on AMR3.0.
  arXiv  Detail & Related papers  (2022-05-03T12:58:36Z)
• Understanding Deformable Alignment in Video Super-Resolution [90.9481291865295]
  We show that deformable convolution can be decomposed into a combination of spatial warping and convolution. We propose an offset-fidelity loss that guides the offset learning with optical flow.
  arXiv  Detail & Related papers  (2020-09-15T17:55:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.