Measuring Alignment-Induced Activation Shifts Correctly: A Template-Controlled Difference-in-Differences Protocol

• URL: http://arxiv.org/abs/2605.24583v3
• Date: Sun, 31 May 2026 07:57:23 GMT
• Title: Measuring Alignment-Induced Activation Shifts Correctly: A Template-Controlled Difference-in-Differences Protocol
• Abstract summary: We show the obvious way to form this matrix is confounded.<n>The aligned model is evaluated under a chat template the base model never saw.<n>We distill it into measurement recommendations for activation-difference studies of alignment.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Comparing a model's internal activations before and after alignment is a natural way to ask what safety training changes: one forms the matrix of paired aligned-minus-base activations on safety-relevant inputs and reads off its effective rank or top direction. We show the obvious way to form this matrix is confounded. The aligned model is evaluated under a chat template the base model never saw, so the naive difference conflates the alignment shift with chat formatting. We introduce a four-variant decomposition of the modification matrix (naive, template-controlled, within-aligned, and difference-in-differences, DiD) that separates the two effects. Template control alone removes a 2.0-3.9x inflation of the measured effective rank across Llama-3.1-8B, Gemma-2-9B, and Qwen-2.5-7B; the DiD contrast is what recovers the refusal direction of Arditi et al. (2024), lifting its cosine alignment from 0.18-0.39 to 0.50-0.86. Projection-ablation across the three families confirms the recovered subspace is behaviorally active and that singular-value order is not causal order. We validate the protocol on a controlled testbed and distill it into measurement recommendations for activation-difference studies of alignment.

Related papers

• SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation [51.46828526392219]
  Reasoning segmentation converts an implicit linguistic conclusion into a precise mask.<n>Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context.<n>We present SeGDeP, an explicit what-where interface.
  arXiv  Detail & Related papers  (2026-09-08T15:11:32Z)
• Activation-Space Order-Swap Geometry: A Site-Asymmetry Audit [0.0]
  Order-dependent activation statistics are often interpreted as evidence of interaction, but that interpretation can be confounded by where interventions enter the network.<n>For a twice-differentiable readout, the open-path order-swap decomposes into a canonical additive response measured by single interventions.<n>Across six open-weight language-model families, the single-intervention baseline explains 84.3-97.7 percent of the bracket norm.
  arXiv  Detail & Related papers  (2026-08-26T02:54:02Z)
• PCA-guided Activation Scaling for Monotonic Bidirectional Control over LLM Sycophancy [15.914340276284085]
  Large language models (LLMs) exhibit sycophancy, a tendency to agree with user beliefs regardless of factual accuracy.<n>We introduce PCA-guided Activation Scaling (PAS), an activation steering framework that decomposes residual stream activations into a PCA-identified sycophancy-honesty subspace.<n>Across three LLMs and three datasets, PAS achieves strong monotonicity (Spearman $$ = +0.92) and an average shift of 15.4% per direction.
  arXiv  Detail & Related papers  (2026-08-17T14:48:49Z)
• Detecting Safety Training Modification in Language Models via Activation Analysis [0.0]
  We introduce AMS, a tool that detects modifications to safety training in language models by measuring the geometric structure of safety-relevant concepts in activation space.<n>We validate AMS across 14 model configurations spanning 4 architecture families.<n>Mechanistic analysis identifies a four-class taxonomy of safety-training modifications distinguished by activation-space signature.
  arXiv  Detail & Related papers  (2026-08-06T03:57:38Z)
• Present but Rescaled: Chat-to-Agent Transfer of Additive Activation Steering [0.7252027234425333]
  We present the first systematic chat-to-agent transfer study of additive steering.<n>The transfer is real but rescaled, and the right description is a dissociation.<n>Agentic deployment amplifies steering-based refusal bypass by up to 2.00x on some models while others.
  arXiv  Detail & Related papers  (2026-07-10T07:21:34Z)
• Actionable Activation Directions for Detecting and Mitigating Emergent Misalignment Across Language Model Families [0.0]
  Fine-tuning language models on insecure code induces emergent misalignment with poorly understood internal structure.<n>We investigate whether this misalignment corresponds to a causally actionable activation-space direction shared across architectures.
  arXiv  Detail & Related papers  (2026-06-18T13:39:59Z)
• CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction [51.56484100374058]
  Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness.<n>We introduce textscCaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction.<n>textscCaliDist consistently achieves lower Expected Error (ECE) and Brier Score compared with strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T07:27:53Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes [4.738949927143789]
  We investigate whether linearly decodable failure signals in hidden states can be leveraged to correct those failures.<n>We find that decodable failure structure supports post-generation reliability estimation even when the fixed linear steering family cannot exploit it for correction.
  arXiv  Detail & Related papers  (2026-05-07T05:58:38Z)
• When Self-Reference Fails to Close: Matrix-Level Dynamics in Large Language Models [0.0]
  Self-referential statements and meta-cognitive prompts are more stable than paradoxical self-reference on key collapse-related metrics.<n>We propose, as a conjecture, that NCTR forces finite-depth transformers toward dynamical regimes where these problems concentrate.
  arXiv  Detail & Related papers  (2026-04-13T23:23:02Z)
• Hierarchical Kernel Transformer: Multi-Scale Attention with an Information-Theoretic Approximation Analysis [0.0]
  The Hierarchical Kernel Transformer (HKT) is a multi-scale attention mechanism that processes sequences at L resolution levels via trainable causal downsampling.<n>The total computational cost is bounded by 4/3 times that of standard attention, reaching 1.3125x for L = 3.
  arXiv  Detail & Related papers  (2026-04-10T00:00:11Z)
• The Geometry of Harmful Intent: Training-Free Anomaly Detection via Angular Deviation in LLM Residual Streams [0.0]
  We present LatentBiopsy, a training-free method for detecting harmful prompts by analysing the geometry of residual-stream activations in large language models.<n>We evaluate two complete model triplets from the Qwen3.5-0.8B and Qwen2.5-0.5B families.<n>Across all six variants, LatentBiopsy achieves AUROC $geq$0.937 for harmful-vs-normative detection and AUROC = 1.000 for discriminating harmful from benign-aggressive prompts.
  arXiv  Detail & Related papers  (2026-03-28T21:19:58Z)
• On the Structural Non-Preservation of Epistemic Behaviour under Policy Transformation [51.56484100374058]
  We formalise such information-conditioned interaction patterns as behavioural dependency.<n>This induces a probe-relative notion of $$-behavioural equivalence and a within-policy behavioural distance.<n>Results identify structural conditions under which probe-conditioned behavioural separation is not preserved under common policy transformations.
  arXiv  Detail & Related papers  (2026-02-24T22:55:21Z)
• Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning [17.384089089363382]
  We identify a root cause that existing methods overlook: the uniform penalization of errors.<n>Current approaches treat all incorrect rollouts within a group identically.<n>We propose the Asymmetric Confidence-aware Error Penalty (ACE)
  arXiv  Detail & Related papers  (2026-02-24T22:46:43Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Why Alignment Must Precede Distillation: A Minimal Working Explanation [50.784080714897776]
  We show that the standard KD -> Align workflow diminishes the model's capacity to align rare yet desirable behaviors.<n>We demonstrate that alignment must first be performed on a high-recall reference before distillation.
  arXiv  Detail & Related papers  (2025-09-28T06:12:19Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs [56.93583799109029]
  GrAInS is an inference-time steering approach that operates across both language-only and vision-language models and tasks.<n>During inference, GrAInS hidden activations at transformer layers guided by token-level attribution signals, and normalizes activations to preserve representational scale.<n>It consistently outperforms both fine-tuning and existing steering baselines.
  arXiv  Detail & Related papers  (2025-07-24T02:34:13Z)
• Krotov Type Optimization of Coherent and Incoherent Controls for Open Two-Qubit Systems [77.34726150561087]
  This work considers two-qubit open quantum systems driven by coherent and incoherent controls. Incoherent control induces time-dependent decoherence rates via time-dependent spectral density of the environment. The system evolves according to the Gorini-Kossakowski-Sudarshan-Lindblad master equation with time-dependent coefficients.
  arXiv  Detail & Related papers  (2023-08-11T13:17:19Z)
• Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation [64.92152574895111]
  We propose a simple Orthogonal Jacobian Regularization (OroJaR) to encourage deep generative model to learn disentangled representations. Our method is effective in disentangled and controllable image generation, and performs favorably against the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-17T15:01:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.