Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training

• URL: http://arxiv.org/abs/2605.06076v1
• Date: Thu, 07 May 2026 11:59:52 GMT
• Title: Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training
• Abstract summary: We systematically track the structural evolution of Transformer circuits throughout the supervised fine-tuning process.<n>Our empirical results reveal that circuits inherently exhibit "Free Evolution" during parameter updates.<n>This work underscores the necessity of "foresight" in mechanistic localization and proposes a framework for future research.
• Score: 59.49057547517285
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The "Locate-then-Update" paradigm has become a predominant approach in the post-training of large language models (LLMs), identifying critical components via mechanistic interpretability for targeted parameter updates. However, this paradigm rests on a fundamental yet unverified assumption: can mechanisms derived from current static parameters reliably guide future dynamic parameter updates? To investigate this, we systematically track the structural evolution of Transformer circuits throughout the supervised fine-tuning (SFT) process, revealing the underlying dynamics of task mechanisms. We introduce three novel metrics-Circuit Distance, Circuit Stability, and Circuit Conflict-to analyze circuit evolution across three dimensions: neural migration, semantic stability, and cross-task interference. Our empirical results reveal that circuits inherently exhibit "Free Evolution" during parameter updates. Consequently, static mechanisms extracted from current states inevitably suffer from temporal latency, making them fundamentally inadequate for guiding future states. Moreover, by deconstructing the "illusion of effectiveness" in existing methods, this work underscores the necessity of "foresight" in mechanistic localization and proposes a predictive framework for future research.

Related papers

• Beyond Static Interpretability: Anticipating Post-SFT Mechanisms from Pre-SFT Parameters for Better Tuning [28.24632224083753]
  Mechanistic localization bridges mechanistic interpretability and post-training optimization.<n>We propose a forward-looking framework that accurately estimates the post-SFT interpretability state.
  arXiv  Detail & Related papers  (2026-08-25T12:28:55Z)
• Variational Parameter Calibration with Physics-Aware Latent-Space Surrogates [3.5829219417440368]
  We introduce a physics-aware neural-network-based latent-space framework for reduced-order forward modeling and variational parameter estimation.<n>The proposed autoencoder-based approach yields a differentiable surrogate that maps physical parameters to predicted flow fields through a latent representation.
  arXiv  Detail & Related papers  (2026-08-11T21:02:31Z)
• Continual Learning in Transition [77.0733079568852]
  Classical continual learning has primarily focused on enabling models to update and retain knowledge through parameter-centric mechanisms.<n>Emerging paradigms are reshaping the scope of CL beyond this traditional model adaptation view.<n>We examine the evolution of continual learning through three dimensions: When, How, and Where learning occurs.
  arXiv  Detail & Related papers  (2026-08-06T16:07:26Z)
• Unlocking Feature Learning in Gated Delta Networks at Scale [52.705562336559446]
  Training and scaling Large Language Models demand enormous computational resources.<n>We derive the scaling rules for Gated Delta Network.<n> Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer.
  arXiv  Detail & Related papers  (2026-06-02T08:45:24Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• KoopGen: Koopman Generator Networks for Representing and Predicting Dynamical Systems with Continuous Spectra [65.11254608352982]
  We introduce a generator-based neural Koopman framework that models dynamics through a structured, state-dependent representation of Koopman generators.<n>By exploiting the intrinsic Cartesian decomposition into skew-adjoint and self-adjoint components, KoopGen separates conservative transport from irreversible dissipation.
  arXiv  Detail & Related papers  (2026-02-15T06:32:23Z)
• From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models [77.04403907729738]
  This survey charts the evolution of uncertainty from a passive diagnostic metric to an active control signal guiding real-time model behavior.<n>We demonstrate how uncertainty is leveraged as an active control signal across three frontiers.<n>This survey argues that mastering the new trend of uncertainty is essential for building the next generation of scalable, reliable, and trustworthy AI.
  arXiv  Detail & Related papers  (2026-01-22T06:21:31Z)
• A Mechanistic Analysis of Transformers for Dynamical Systems [4.590170084532207]
  We study the representational capabilities and limitations of single-layer Transformers when applied to dynamical data.<n>For linear systems, we show that the convexity constraint imposed by softmax attention fundamentally restricts the class of dynamics that can be represented.<n>For nonlinear systems under partial observability, attention instead acts as an adaptive delay-embedding mechanism.
  arXiv  Detail & Related papers  (2025-12-24T11:21:07Z)
• Benchmarking neural surrogates on realistic spatiotemporal multiphysics flows [18.240532888032394]
  We present REALM (REalistic AI Learning for Multiphysics), a rigorous benchmarking framework designed to test neural surrogates on challenging, application-driven reactive flows.<n>We benchmark over a dozen representative surrogate model families, including spectral operators, convolutional models, Transformers, pointwise operators, and graph/mesh networks.<n>We identify three robust trends: (i) a scaling barrier governed jointly by dimensionality, stiffness, and mesh irregularity, leading to rapidly growing rollout errors; (ii) performance primarily controlled by architectural inductive biases rather than parameter count; and (iii) a persistent gap between nominal accuracy metrics and physically
  arXiv  Detail & Related papers  (2025-12-21T05:04:13Z)
• Time-Scale Coupling Between States and Parameters in Recurrent Neural Networks [3.924071936547547]
  Gated neural networks (RNNs) implicitly induce adaptive learning-rate behavior.<n>Effect arises from the coupling between state-space time scales--parametrized by the gates--and parameter-space dynamics.<n> Empirical simulations corroborate these claims.
  arXiv  Detail & Related papers  (2025-08-16T18:19:34Z)
• Forecasting Continuous Non-Conservative Dynamical Systems in SO(3) [51.510040541600176]
  We propose a novel approach to modeling the rotation of moving objects in computer vision.<n>Our approach is agnostic to energy and momentum conservation while being robust to input noise.<n>By learning to approximate object dynamics from noisy states during training, our model attains robust extrapolation capabilities in simulation and various real-world settings.
  arXiv  Detail & Related papers  (2025-08-11T09:03:10Z)
• UP-dROM : Uncertainty-Aware and Parametrised dynamic Reduced-Order Model, application to unsteady flows [27.50487430169627]
  Reduced order models (ROMs) play a critical role in fluid mechanics by providing low-cost predictions.<n>For ROMs to be widely applicable, they must not only generalise well across different regimes, but also provide a measure of confidence in their predictions.<n>We present a nonlinear reduction strategy specifically designed for transient flows.
  arXiv  Detail & Related papers  (2025-03-29T22:17:36Z)
• Fine-Tuning is Subgraph Search: A New Lens on Learning Dynamics [13.58024883022897]
  We develop a fine-tuning method for analyzing the mechanism behind learning.<n>Inspired by the concept of intrinsic dimension, we view a model as a computational graph with redundancy for a specific task.<n>We propose circuit-tuning, an algorithm that iteratively builds the subgraph for a specific task and updates the relevant parameters in a way.
  arXiv  Detail & Related papers  (2025-02-10T02:35:53Z)
• Supporting Optimal Phase Space Reconstructions Using Neural Network Architecture for Time Series Modeling [68.8204255655161]
  We propose an artificial neural network with a mechanism to implicitly learn the phase spaces properties. Our approach is either as competitive as or better than most state-of-the-art strategies.
  arXiv  Detail & Related papers  (2020-06-19T21:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.