Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation

• URL: http://arxiv.org/abs/2605.07302v1
• Date: Fri, 08 May 2026 06:12:43 GMT
• Title: Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation
• Abstract summary: Finetuning pretrained models occurs in a low-dimensional subspace of the full parameter space.<n>Are stable directions irrelevant to downstream tasks, or do they already encode task-relevant structure that requires no further adjustment?<n>We show that the leading singular vectors of pretrained weight matrices remain highly stable under finetuning and are shared across unrelated downstream tasks.
• Score: 10.547646302449682
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Finetuning pretrained models occurs in a low-dimensional subspace of the full parameter space. Prior work has focused on characterizing this optimization subspace, but largely ignored the complementary question: why do certain directions remain unexplored during finetuning? Are these stable directions irrelevant to downstream tasks, or do they already encode task-relevant structure that requires no further adjustment? Answering this question is central to understanding how pretrained knowledge transfers. Through systematic spectral analysis across vision and language models, we show that the leading singular vectors of pretrained weight matrices remain highly stable under finetuning and are shared across unrelated downstream tasks, revealing that pretraining establishes a reusable spectral coordinate system. Models pretrained on larger datasets exhibit greater spectral stability under distribution shift or task change, directly linking pretraining scale to geometric transferability. Motivated by these findings, we propose a parameter-efficient method that freezes pretrained singular vectors and optimizes only leading spectral coefficients, achieving competitive performance on GLUE with 0.2% trainable parameters. Our results reveal that the stable directions encode transferable structure rather than irrelevant noise: successful pretraining discovers spectral bases that downstream tasks inherit and operate within.

Related papers

• Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent [18.338094508559468]
  Under a fixed training budget, compute spent improving the upstream objective reduces the compute available for downstream adaptation.<n>We cast this allocation as a compute-split problem under a two-stage pretrain--fine-tune procedure with fixed total optimisation budget.<n>Our results show that the allocation depends on how pretraining directions affect fine-tuning predictions and how fine-tuning shifts are seen through downstream data geometry.
  arXiv  Detail & Related papers  (2026-09-14T19:23:59Z)
• Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation [5.220809937383775]
  New-task training induces old-task prediction drift through the cross-task kernel.<n>In frozen-backbone linear-head PEFT-CL, where the model is linear in the trainable parameters, the predictor is exact up to numerical precision.
  arXiv  Detail & Related papers  (2026-06-16T15:03:37Z)
• Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer [13.823003260600663]
  When a new version of a pre-trained model becomes available, expertise acquired through fine-tuning cannot be directly reused.<n>We propose BiCo, a training-free framework for transferring task vectors through Bilinear Coordinate alignment.<n>BiCo consistently outperforms existing transfer methods across models that differ in width, depth, and pre-training configuration.
  arXiv  Detail & Related papers  (2026-05-27T13:10:34Z)
• Rank Is Not Capacity: Spectral Occupancy for Latent Graph Models [8.608322716392232]
  We propose Spectral Prefix Extraction and Capacity-Targeted Representation Analysis (Spectra)<n>We replace rank as the unit of analysis with the spectrum of a learned positive semidefinite kernel, trace-normalized so that spectra are comparable across fits.<n>Spectra traces performance--capacity frontiers that make the trade-off between predictive accuracy and realized dimension visible.
  arXiv  Detail & Related papers  (2026-05-11T18:46:53Z)
• SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Training [70.84726713548099]
  SAE-based Transferability Score (STS) is a new metric to forecast post-training transferability.<n>We show that STS accurately predicts the transferability of supervised fine-tuning, achieving Pearson correlation coefficients above 0.7 with actual performance changes.
  arXiv  Detail & Related papers  (2026-03-03T12:01:09Z)
• Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation [58.3773038915023]
  Continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge.<n>Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge.<n>We address this problem using a projected first-order method compatible with standard deep-dots used in vision-language models.
  arXiv  Detail & Related papers  (2026-01-31T13:27:02Z)
• Basis-Oriented Low-rank Transfer for Few-Shot and Test-Time Adaptation [10.804106052326402]
  Adapting large pre-trained models to unseen tasks under tight data and compute budgets remains challenging.<n>We propose BOLT, a framework that reuses existing fine-tuned models and adapts within that subspace.<n>Our results show that constraining adaptation to a task-informed subspace provides an effective alternative for unseen-task transfer.
  arXiv  Detail & Related papers  (2025-12-02T06:00:16Z)
• Weight Spectra Induced Efficient Model Adaptation [54.8615621415845]
  Fine-tuning large-scale foundation models incurs prohibitive computational costs.<n>We show that fine-tuning predominantly amplifies the top singular values while leaving the remainder largely intact.<n>We propose a novel method that leverages learnable rescaling of top singular directions.
  arXiv  Detail & Related papers  (2025-05-29T05:03:29Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets [74.11825654535895]
  Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance. We study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
  arXiv  Detail & Related papers  (2021-09-08T10:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.