Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation

• URL: http://arxiv.org/abs/2609.00762v1
• Date: Tue, 01 Sep 2026 05:52:38 GMT
• Title: Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation
• Abstract summary: We study the choice of constructor-efficient fine-tuning under a severe trainable-state budget.<n>We introduce FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates.<n>On Qwen2.5-3B, FCCA reaches an 83.0 macro-average, 2.3 points above the next-best matched-budget constructor, and exceeds its unwhitened RawGrad control on all 11 tasks.
• Score: 18.8104643146881
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential. We study this choice through frozen-core adaptation: a calibration pass fixes left and right bases for each weight matrix, and fine-tuning optimizes only an $r\times r$ core. This removes the ability of trainable factors to repair a poor initial span and makes subspace quality directly observable. We introduce FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates. Under a matched $r^2$ budget, we compare eight basis constructors on 11 tasks, four model settings, and three seeds. On Qwen2.5-3B, FCCA reaches an 83.0 macro-average, 2.3 points above the next-best matched-budget constructor, and exceeds its unwhitened RawGrad control on all 11 tasks. It ranks first at all three Qwen scales and finishes within 0.13 points of the best method on Llama-3.2-1B. Controlled ablations show gains of 2.7--17.2 points from whitening and identify QR as necessary for stable core optimization in the tested regime. Finally, FCCA comes within 0.32 and 0.23 average points of LoRA and DoRA while optimizing 36.9K rather than roughly 7.4M parameters. These results show that a carefully selected fixed span can recover most of the benefit of movable low-rank factors at a much smaller trainable and optimizer-state cost.

Related papers

• EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale [9.562123195246393]
  EGGROLL makes evolution strategies practical for LLMs by replacing dense Gaussian weight perturbations with low-rank Gaussian products.<n>We analyze the error of its finite-population estimator.<n>LOO-ROLL, a leave-one-out estimator, preserves the finite-rank population field while replacing EGGROLL's two antithetic evaluations per direction by one.
  arXiv  Detail & Related papers  (2026-09-10T01:58:02Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning [70.71795707610961]
  method augments SOAP-style preconditioning with a scalar secant-energy correction adapted to Kronecker geometry.<n>Method attains the lowest final residual on six of eight PDE benchmarks.<n>Three-seed $L2$ and $H1$ errors on four representative PDEs support the link between lower residuals and improved solution accuracy.
  arXiv  Detail & Related papers  (2026-08-29T21:34:47Z)
• SchurQuant: Groupwise Discrete Optimization for Layer-Wise LLM Quantization [9.043029012704817]
  We introduce SCHUROPT, which analytically eliminates the suffix's optimal continuous response.<n>It alternates closed-form row-wise scale/zero-point refitting with coordinate descent over integer codes.<n>Across eight Llama and Qwen models, SCHURQUANT achieves the highest mean zero-shot accuracy.
  arXiv  Detail & Related papers  (2026-08-16T06:29:10Z)
• Agentic Stage-One Stellarator Optimization: Autonomous Multi-Objective Search for Finite-Beta Equilibria [0.0]
  We present a proof of concept for emphagentic stage-one optimization.<n>Agentic outer-loop control can sustain finite-beta, multi-objective search and turn repeated optimization into a scalable source of improved equilibria.
  arXiv  Detail & Related papers  (2026-08-02T16:08:52Z)
• TabPFN beyond Tabular Data: Calibration and Accuracy on Multimodal Embeddings [2.4483752182160954]
  We evaluate TabPFN as a plug-and-play, zero-gradient classification head for frozen image, text, and audio encoders.<n>Across 22,820 evaluation episodes spanning 14 datasets, 11 encoders, and three modalities, TabPFN achieves the best mean rank among nine classification heads on both negative log-likelihood (NLL) and expected calibration error (ECE)<n>Its accuracy advantage is conditional, concentrating at moderate-to-high shot counts and low-to-moderate feature dimensions.
  arXiv  Detail & Related papers  (2026-07-13T02:11:27Z)
• Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study [48.83701310501069]
  This paper investigates whether a single recipe is sufficient for heterogeneous architecture pools, or whether the optimal pairing varies across structurally diverse models.<n>We conduct a systematic empirical study of all $3 times 6 = 18$ combinations of six classifiers paired with three loss functions: Cross-Entropy (CEL), Negative Log-Likelihood (NLL), and the recently introduced genetically evolved NGL loss across the base models presented in LEMUR heterogeneous architecture pool on six image classification datasets.<n>Our results confirm that no single pairing is universally optimal. Cross-Entropy with Adam or AdamW is the most robust choice across
  arXiv  Detail & Related papers  (2026-06-18T20:51:42Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Step-wise Rubric Rewards for LLM Reasoning [72.17879367869503]
  Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models.<n>We find 18.2% of steps in correct responses are wrong yet positively rewarded.<n>Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points.
  arXiv  Detail & Related papers  (2026-05-17T07:08:14Z)
• One-for-All: A Lightweight Stabilized and Parameter-Efficient Pre-trained LLM for Time Series Forecasting [4.364999214109123]
  One-for-All is a framework for adapting pre-trained Large Language Models for time-series analysis.<n>rsLoRA introduces a mathematically grounded rank-stabilization mechanism that enables provable gradient stability at low ranks.<n>One-for-All achieves state-of-the-art efficiency-accuracy trade-offs.
  arXiv  Detail & Related papers  (2026-03-31T13:54:43Z)
• CarBoN: Calibrated Best-of-N Sampling Improves Test-time Reasoning [62.56541355300587]
  We introduce a general test-time calibration framework that adaptively modifies the model toward high-reward reasoning paths.<n>Within this framework, we propose CarBoN, a two-phase method that first explores the solution space and then learns a calibration of the logits.<n>Experiments on MATH-500 and AIME-2024 show that CarBoN improves efficiency, with up to $4times$ fewer rollouts to reach the same accuracy.
  arXiv  Detail & Related papers  (2025-10-17T14:04:37Z)
• Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processes [80.89852729380425]
  We propose the first computationally efficient algorithm that achieves the nearly minimax optimal regret $tilde O(dsqrtH3K)$. Our work provides a complete answer to optimal RL with linear MDPs, and the developed algorithm and theoretical tools may be of independent interest.
  arXiv  Detail & Related papers  (2022-12-12T18:58:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.