First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

• URL: http://arxiv.org/abs/2607.16821v1
• Date: Sat, 18 Jul 2026 13:43:44 GMT
• Title: First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers
• Abstract summary: We measure 8 properties with the same harness around a multitask LoRA operating point.<n>We find a shared one-direction validity window up to the tested scale $10-2$, but no universal radius for pairwise composition or update ordering.
• Score: 0.05729426778193398
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space. We measure 8 such properties with the same harness around a multitask LoRA operating point, on 9 transformers (82M-7B), with a prospectively registered property list, thresholds, and test split. We find a shared one-direction validity window up to the tested scale $10^{-2}$, but no universal radius for pairwise composition or update ordering. Along individual directions, changes of the probe loss remain first-order predictable throughout the grid: a perturbation's effect on the loss is essentially its projection onto the gradient, which is also what makes local random search work. Pairwise structure, however, proves to be far more fragile: on over a third of the measured (model, task pair) combinations, two-update order sensitivity sets in strictly inside that window; task-gradient subspaces rotate within tens of steps; additivity under our fixed activation probe fails at full task-vector scale on several models, including both held-out 7B models; and no model median passes the registered global mean-vector weight-to-steering correspondence bar. For two sequential task-gradient steps, the leading order-dependent term is the Lie bracket $H_B\textbf{g}_A-H_A\textbf{g}_B$; its normalized prediction $c(η)=ηκ+O(η^2)$ tracks the measured defect at median ratio 1.002, while the onset scale $η^\dagger\approx0.10/κ$ spans three orders of magnitude across models and task pairs.

Related papers

• Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation [18.8104643146881]
  We study the choice of constructor-efficient fine-tuning under a severe trainable-state budget.<n>We introduce FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates.<n>On Qwen2.5-3B, FCCA reaches an 83.0 macro-average, 2.3 points above the next-best matched-budget constructor, and exceeds its unwhitened RawGrad control on all 11 tasks.
  arXiv  Detail & Related papers  (2026-09-01T05:52:38Z)
• Steer the Sampling, Not the Kernel Grid: Geometry-Guided Sampling Operator for Volumetric Segmentation [9.411407984734854]
  We propose a geometry-guided local operator that steers where features are sampled, rather than deforming convolutional kernels.<n>At each voxel, it predicts a local orientation and bounded step sizes, samples symmetrically along these directions.<n> Replacing all stride 1 and stride 2 operators in a 3D U-Net yields consistent improvements on BraTS, MSD Hepatic Vessel, and TDSC-ABUS.
  arXiv  Detail & Related papers  (2026-08-26T14:01:36Z)
• When Do Task Vectors Interfere? Mapping the Validity Boundaries of Weight-Space Composition [3.2114065314849323]
  Task arithmetic treats fine-tuning displacements as composable directions in weight space.<n>We separate parameter geometry from functional geometry and measure pairwise functional non-additivity over a two-dimensional task-vector surface.
  arXiv  Detail & Related papers  (2026-08-10T11:58:04Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• The Geometric Canary: Predicting Steerability and Detecting Drift via Representational Stability [0.0]
  We show that geometric stability, the consistency of a representation's pairwise distance structure, addresses two functions.<n> unsupervised stability fails entirely for steering on real-world tasks.<n>It excels at drift detection, measuring nearly $2times$ greater geometric change than CKA during post-training alignment.
  arXiv  Detail & Related papers  (2026-04-20T01:24:45Z)
• Anatomy of Capability Emergence: Scale-Invariant Representation Collapse and Top-Down Reorganization in Neural Networks [1.5567685129899713]
  We track five geometric measures across five model scales.<n>We find that training begins with a universal representation collapse to task-specific floors.<n>On Pythia, global geometric patterns replicate but per-task precursor signals do not.
  arXiv  Detail & Related papers  (2026-02-17T20:39:02Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models [50.11453013647086]
  We analyze the forgetting, i.e., loss on previously seen tasks, after $k$ iterations.<n>We develop novel last-iterate upper bounds in the realizable least squares setup.<n>We prove for the first time that randomization alone, with no task repetition, can prevent catastrophic in sufficiently long task sequences.
  arXiv  Detail & Related papers  (2025-04-06T18:39:45Z)
• AffineGlue: Joint Matching and Robust Estimation [74.04609046690913]
  We propose AffineGlue, a method for joint two-view feature matching and robust estimation. AffineGlue selects potential matches from one-to-many correspondences to estimate minimal models. Guided matching is then used to find matches consistent with the model, suffering less from the ambiguities of one-to-one matches.
  arXiv  Detail & Related papers  (2023-07-28T08:05:36Z)
• Semi-DETR: Semi-Supervised Object Detection with Detection Transformers [105.45018934087076]
  We analyze the DETR-based framework on semi-supervised object detection (SSOD) We present Semi-DETR, the first transformer-based end-to-end semi-supervised object detector. Our method outperforms all state-of-the-art methods by clear margins.
  arXiv  Detail & Related papers  (2023-07-16T16:32:14Z)
• Hausdorff Distance Matching with Adaptive Query Denoising for Rotated Detection Transformer [4.137346786534721]
  We introduce a Hausdorff distance-based cost for bipartite matching, which more accurately quantifies the discrepancy between predictions and ground truths. We propose an adaptive query denoising method that employs bipartite matching to selectively eliminate noised queries that detract from model improvement.
  arXiv  Detail & Related papers  (2023-05-12T16:42:54Z)
• Analyzing and Mitigating Interference in Neural Architecture Search [96.60805562853153]
  We investigate the interference issue by sampling different child models and calculating the gradient similarity of shared operators. Inspired by these two observations, we propose two approaches to mitigate the interference. Our searched architecture outperforms RoBERTa$_rm base$ by 1.1 and 0.6 scores and ELECTRA$_rm base$ by 1.6 and 1.1 scores on the dev and test set of GLUE benchmark.
  arXiv  Detail & Related papers  (2021-08-29T11:07:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.