Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning

• URL: http://arxiv.org/abs/2606.07631v1
• Date: Sun, 31 May 2026 04:28:21 GMT
• Title: Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning
• Abstract summary: Emergent misalignment (EM) occurs when a model behaves dangerously outside the finetuning task.<n>Standard training signals can miss this shift, making reliable detection costly if it depends on repeated behavioral evaluation.<n>We ask whether emergent misalignment can instead be detected from internal representations during finetuning.
• Score: 13.759856460598087
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Emergent misalignment (EM) occurs when narrow finetuning causes a model to behave dangerously outside the finetuning task. Standard training signals can miss this shift, making reliable detection costly if it depends on repeated behavioral evaluation. We ask whether emergent misalignment can instead be detected from internal representations during finetuning. Using seven alignment-relevant traits encoded as linear directions in activation space, we track representational drift across training checkpoints in four open-source 7-9B LLMs. EM-relevant drift concentrates on a low-dimensional axis that explains 65.5% of the variance, revealing a geometric signature in the studied regime. A low-overhead monitor built on this drift profile detects dangerous checkpoints with 2.2% false negative rate, 2.9% false positive rate, and 0.990 AUROC on held-out perturbation types, outperforming unsupervised PCA and SAE baselines. Stress tests on two 14B models, longer finetuning runs, and misaligned starting points identify key deployment boundaries. These results position trait-space monitoring as a practical complement to behavioral evaluation for EM detection during LoRA-based finetuning, while showing that deployment across substantially different regimes may require recalibration.

Related papers

• An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon? [7.43544515207381]
  We study repeated alignment and misalignment cycles using controlled fine-tuning loops while tracking behavioral performance, and LoRA representations throughout training.<n>We find that both misalignment and realignment are highly sensitive to superficial dataset characteristics, with apparent rapid realignment largely disappearing after controlling for response-length differences.<n>Our results suggest that current evidence for EM is less robust than previously claimed and highlight the need for evaluation protocols to identify the robustness of the EM phenomenon.
  arXiv  Detail & Related papers  (2026-07-10T02:50:21Z)
• Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning [48.438568700670835]
  The standard way to read latent knowledge out of a model can systematically overstate what a vision-language model grounds in the image.<n>We show this on spatial reasoning, where the error is invisible to both probing and steering yet exposed by a one-line causal control.
  arXiv  Detail & Related papers  (2026-06-30T07:33:18Z)
• Probing the Misaligned Thinking Process of Language Models [32.29324298373053]
  Large language models exhibit a growing range of misaligned behaviors.<n>It is critical to reliably detect such behaviors to ensure safe and responsible use.<n>We propose to monitor misalignment by decomposing it into fine-grained cognitive processes.
  arXiv  Detail & Related papers  (2026-06-23T07:40:28Z)
• Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluation [74.17379276939599]
  We show that activation steering can induce broad misalignment, even in the recent Qwen-3.5 series.<n>We characterize properties of AS-induced EM by analyzing key steering-specific factors, including steering magnitude, the low-rank structure of the steering subspace, and the number of epochs during steering-vector construction.
  arXiv  Detail & Related papers  (2026-06-07T15:34:59Z)
• CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction [51.56484100374058]
  Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness.<n>We introduce textscCaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction.<n>textscCaliDist consistently achieves lower Expected Error (ECE) and Brier Score compared with strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T07:27:53Z)
• Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment [12.66555363236598]
  Emergent misalignment (EM) has been linked to the activation of misaligned persona vectors and evil character traits.<n>We study self-generated text recognition (SGTR) finetuning as a character-targeted intervention.
  arXiv  Detail & Related papers  (2026-06-04T00:04:58Z)
• Manifold-Guided Attention Steering [23.05277076113183]
  We propose a trajectory-aware inference-time intervention grounded in a geometric observation.<n>The output activations of specific attention heads diverge from a low-dimensional correctness manifold at the point of error.<n>We apply a targeted projection correction when deviation exceeds a learned threshold, steering the attention output back toward the correct subspace before the error propagates.
  arXiv  Detail & Related papers  (2026-05-20T22:06:08Z)
• TACT: Mitigating Overthinking and Overacting in Coding Agents via Activation Steering [70.99933391739154]
  We focus on two recurring failure modes where the agent repeatedly reasons over information it already has, and where it issues tool calls without integrating recent observations or acquiring new evidence.<n>In this paper, we introduce TACT (Think-Act via activation Steering), to detect and mitigate agent drift in the residual stream before it surfaces as a behavioral failure.<n>In specific, we label trajectory steps as overthinking, overacting, or calibrated, and find that their hidden states can separate linearly along two *drift axes*, pointing from calibrated behavior toward each failure mode.
  arXiv  Detail & Related papers  (2026-05-07T10:24:27Z)
• Mitigating Error Amplification in Fast Adversarial Training [58.74042726356826]
  Fast Adversarial Training (FAT) has proven effective in enhancing model robustness by encouraging networks to learn perturbation-invariant representations.<n>FAT often suffers from catastrophic overfitting (CO), where the model overfits to the training attack and fails to generalize to unseen ones.<n>We propose a Distribution-aware Dynamic Guidance (DDG) strategy that dynamically adjusts both the perturbation budget and supervision signal.
  arXiv  Detail & Related papers  (2026-04-27T11:23:18Z)
• Evidence Sufficiency Under Delayed Ground Truth: Proxy Monitoring for Risk Decision Systems [0.0]
  This paper formalizes an evidence sufficiency model with four dimensions (completeness, freshness, reliability, representativeness) and a decision-readiness gate.<n>A complementary proxy indicator framework estimates sufficiency degradation without labels.<n>The framework contributes a governance sufficiency monitoring instrument.
  arXiv  Detail & Related papers  (2026-04-17T06:30:11Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• Noise & pattern: identity-anchored Tikhonov regularization for robust structural anomaly detection [58.535473924035365]
  Anomaly detection plays a pivotal role in automated industrial inspection, aiming to identify subtle or rare defects in otherwise uniform visual patterns.<n>We tackle structural anomaly detection using a self-supervised autoencoder that learns to repair corrupted inputs.<n>We introduce a corruption model that injects artificial disruptions into training images to mimic structural defects.
  arXiv  Detail & Related papers  (2025-11-10T15:48:50Z)
• Dual-Mode Deep Anomaly Detection for Medical Manufacturing: Structural Similarity and Feature Distance [2.467339701756281]
  We propose two attention-guided autoencoder architectures for deep anomaly detection.<n>The first employs a structural similarity-based scoring approach that enables lightweight, real-time defect detection.<n>The second applies a feature distance-based strategy using Mahalanobis scoring on reduced latent features.
  arXiv  Detail & Related papers  (2025-09-06T18:17:40Z)
• Hysteresis Compensation of Flexible Continuum Manipulator using RGBD Sensing and Temporal Convolutional Network [2.387821008001523]
  Cable-driven manipulators face control difficulties due to from cabling effects such as friction, elongation, and coupling. This paper proposes a data-driven approach based on Deep Neural Networks (DNN) to capture these nonlinear and previous states-dependent characteristics. Applying this method in real surgical scenarios has the potential to enhance control precision and improve surgical performance.
  arXiv  Detail & Related papers  (2024-02-17T16:20:59Z)
• Detecting Rewards Deterioration in Episodic Reinforcement Learning [63.49923393311052]
  In many RL applications, once training ends, it is vital to detect any deterioration in the agent performance as soon as possible. We consider an episodic framework, where the rewards within each episode are not independent, nor identically-distributed, nor Markov. We define the mean-shift in a way corresponding to deterioration of a temporal signal (such as the rewards), and derive a test for this problem with optimal statistical power.
  arXiv  Detail & Related papers  (2020-10-22T12:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.