Beyond Static Interpretability: Anticipating Post-SFT Mechanisms from Pre-SFT Parameters for Better Tuning

• URL: http://arxiv.org/abs/2608.24482v1
• Date: Tue, 25 Aug 2026 12:28:55 GMT
• Title: Beyond Static Interpretability: Anticipating Post-SFT Mechanisms from Pre-SFT Parameters for Better Tuning
• Abstract summary: Mechanistic localization bridges mechanistic interpretability and post-training optimization.<n>We propose a forward-looking framework that accurately estimates the post-SFT interpretability state.
• Score: 28.24632224083753
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mechanistic Localization bridges mechanistic interpretability and post-training optimization by isolating critical parameters via interpretative approaches and then guiding parameter-efficient Supervised Fine-Tuning (SFT) in a ``locating-then-tuning'' paradigm. However, due to the retrospective nature of mechanistic interpretability, directly interpreting pre-SFT models introduces misleading conclusions. Specifically for novel tasks, initially identified neurons differ drastically from those governing the final model, introducing biases that actively disrupt SFT. To address this, we propose a forward-looking localization framework that accurately estimates the post-SFT interpretability state using only pre-SFT parameters and the target dataset. Theoretically, we model SFT as a continuous parameter evolution, leveraging Taylor expansion to rigorously bridge the post-tuning mechanistic objective with the pre-SFT model's dynamic gradients. Practically, we design dual-granularity (neuron- and component-level) localization pipelines. Extensive experiments demonstrate that our approach not only provides superior SFT guidance but also exhibits robust performance and temporal scalability across increasing model sizes. This work transcends the fundamental limitation of traditional interpretability-its inability to identify task-critical mechanisms before they are trained-pioneering a predictive frontier that unites mechanistic interpretability with targeted optimization.

Related papers

• LP-SFT: Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure [16.743783987674842]
  Supervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains.<n>We propose LP-SFT, a Local-Preserving Supervised Fine-Tuning objective designed to explicitly protect this inherent entropy structure.
  arXiv  Detail & Related papers  (2026-07-06T07:14:22Z)
• Unlocking Feature Learning in Gated Delta Networks at Scale [52.705562336559446]
  Training and scaling Large Language Models demand enormous computational resources.<n>We derive the scaling rules for Gated Delta Network.<n> Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer.
  arXiv  Detail & Related papers  (2026-06-02T08:45:24Z)
• PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective [52.693471818837395]
  We argue that PEFT should be assessed through the stability-plasticity dilemma.<n>We introduce PEFT-Arena, a benchmark that measures downstream performance and general capability retention.<n>Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.
  arXiv  Detail & Related papers  (2026-05-27T17:59:51Z)
• Hyperparameter Trajectory Inference with Conditional Lagrangian Optimal Transport [51.56484100374058]
  Post-deployment, user preferences can evolve, making initial settings undesirable.<n>We learn, from observed data, how a NN's conditional output distribution changes with its hyper parameters.<n>We construct a surrogate model that approximates the NN at unobserved hyper parameters.
  arXiv  Detail & Related papers  (2026-03-02T11:55:02Z)
• Optimizing Multi-Modal Trackers via Sensitivity-aware Regularized Tuning [112.12667472919723]
  This paper tackles the challenge of optimizing multi-modal trackers by effectively adapting the pre-trained models for RGB data.<n>Existing fine-tuning paradigms oscillate between excessive freedom and over-restriction, leading to a suboptimal plasticity-stability trade-off.<n>We propose a novel sensitivity-aware regularized tuning framework, which delicately refines the learning process by incorporating intrinsic parameter sensitivities.
  arXiv  Detail & Related papers  (2025-08-24T18:42:47Z)
• Learnable Adaptive Time-Frequency Representation via Differentiable Short-Time Fourier Transform [11.05158127763157]
  We propose a differentiable formulation of the STFT that enables gradient-based optimization of its parameters.<n>Our approach integrates seamlessly with neural networks, allowing joint optimization of the STFT parameters and network weights.<n>The efficacy of the proposed differentiable STFT in enhancing TFRs and improving performance in downstream tasks is demonstrated.
  arXiv  Detail & Related papers  (2025-06-26T16:24:27Z)
• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections [65.36449542323277]
  We present a unified theoretical framework bridgingSupervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training.<n>We propose a simple yet effective learning rate reduction approach that yields significant performance improvements.
  arXiv  Detail & Related papers  (2025-06-15T05:42:29Z)
• Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models [12.500777267361102]
  We introduce a novel textbfpreference-textbforiented supervised textbffine-textbftuning approach, namely PoFT.<n>The intuition is to boost SFT by imposing a particular preference: textitfavoring the target model over aligned LLMs on the same SFT data.<n>PoFT achieves stable and consistent improvements over the SFT baselines across different training datasets and base models.
  arXiv  Detail & Related papers  (2024-12-17T12:49:14Z)
• Forecast-PEFT: Parameter-Efficient Fine-Tuning for Pre-trained Motion Forecasting Models [68.23649978697027]
  Forecast-PEFT is a fine-tuning strategy that freezes the majority of the model's parameters, focusing adjustments on newly introduced prompts and adapters. Our experiments show that Forecast-PEFT outperforms traditional full fine-tuning methods in motion prediction tasks. Forecast-FT further improves prediction performance, evidencing up to a 9.6% enhancement over conventional baseline methods.
  arXiv  Detail & Related papers  (2024-07-28T19:18:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.