Orientation, not magnitude: the causal structure of task-vector interference in merged language models

• URL: http://arxiv.org/abs/2608.11797v1
• Date: Wed, 12 Aug 2026 08:40:24 GMT
• Title: Orientation, not magnitude: the causal structure of task-vector interference in merged language models
• Abstract summary: We show that task arithmetic works until it doesn't, and the field diagnoses why with magnitudes.<n>An exact decomposition of the layerwise flux shows it is dominated by amplifying transport of the existing cross-term.<n>The striking +-15% "universality" of naive bfloat16 generation turns out to be quantization roughness.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model merging by task arithmetic works until it doesn't, and the field diagnoses why with magnitudes: layerwise representation bias, deviations from cross-task linearity, parameter overlap. Tracking the exact layerwise cross-term of merged LLMs through a factorial ledger and intervening on it directly, we find magnitude insufficient - and inconsistent across model families - as a diagnostic axis. An exact decomposition of the layerwise flux shows it is dominated by amplifying transport of the existing cross-term (~65-70% in both families, gain >1 per late block), and erasing the term is undone by propagation - rebuilt to 99% of its norm at cosine 0.99 - unless applied near the output; a basin test with six starting displacements establishes the carried direction as an attractor of the forward pass. That direction is causally load-bearing: erasure along it removes expressed interference dose-dependently and saturates at exact erasure, while norm-matched wrong-direction controls fail or backfire. Instruction wrappers gate the effect: the same erasure finds 13x less relative interference to remove under a wrapper that internally amplifies the cross-term, because the wrapper drowns the interaction in a template-pinned main effect rather than shrinking it - a structure that replicates across further instruction templates but not under a length-matched control. Magnitude, by contrast, is at best a coarse correlate, and the striking +-15% "universality" of naive bfloat16 generation turns out to be quantization roughness. Task pairs whose local cross-term generation differs by at most 1.9x differ by 14x-337x in causally removable interference. All 46 predictions were preregistered and frozen before their data; falsifications, including of our own headline expectations and of behavioral recovery under a validated continuous endpoint, are reported as such.

Related papers

• Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and a Tight Univariate Rate [47.36630149538994]
  In high-dimensional online prediction, the best predictor may depend on only a few features, so regret should scale with sparsity rather than the ambient dimension.<n>Warmuth and Amid asked at COLT 2023 whether any of three such rules admits a competitive online regret guarantee.<n>Our analysis identifies a common obstruction: cheap nuisance causes the refit to underweight the truly predictive coordinate.
  arXiv  Detail & Related papers  (2026-08-18T09:32:03Z)
• Is Convergence Inevitable? Tracing Output Homogeneity Back to Base Models [49.46165710676499]
  We find that semantic convergence is observed from the first alignment stage.<n>We find that convergence can be revealed and amplified, but not introduced by the SFT data.<n> Taken together, our results suggest that semantic convergence may arise naturally from the objectives underlying LM training.
  arXiv  Detail & Related papers  (2026-08-11T20:47:06Z)
• Do Active SAE Feature Planes Carry More Holonomy? A Preregistered Reversal in Gemma [0.0]
  This paper tests whether holonomy concentrates on active sparse-autoencoder (SAE) feature planes in Gemma 2 2B.<n>The prediction was falsified in reverse: active-feature planes carried less holonomy than matched mixed-feature controls.<n>The cause remains open, with activation-strength geometry, degree of feature engagement, dictionary geometry, matched-center displacement, activation-manifold proximity, and transport shear as live alternatives.
  arXiv  Detail & Related papers  (2026-07-09T05:04:25Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• Actionable Activation Directions for Detecting and Mitigating Emergent Misalignment Across Language Model Families [0.0]
  Fine-tuning language models on insecure code induces emergent misalignment with poorly understood internal structure.<n>We investigate whether this misalignment corresponds to a causally actionable activation-space direction shared across architectures.
  arXiv  Detail & Related papers  (2026-06-18T13:39:59Z)
• GEMS: Geometric Constraints Enable Multi-Semantic Superposition in LLMs [5.018957704056544]
  Activation steering controls model behavior by modifying intermediate hidden states at inference time without retraining.<n>We show that when multiple semantic directions are superposed without constraints, the model collapses.<n>We propose GEMS, a training-free method that maps each source to a corresponding geometric constraint.
  arXiv  Detail & Related papers  (2026-06-18T08:43:33Z)
• Flow Map Denoisers: Traversing the Distortion-Perception Plane for Inverse Problems [48.501415601682815]
  Image restoration faces a fundamental tradeoff: methods that minimize error produce blurry reconstructions, while those that maximize perceptual quality yield sharp but less faithful images.<n>We show that flow map models implicitly define a one- parameter family of denoisers that continuously spans the distortion perception frontier.<n>Within a Plug-and-Play solver, the same mechanism extends to general inverse problems, where it controls a tradeoff between perceptual alignment and data consistency.
  arXiv  Detail & Related papers  (2026-06-18T05:15:43Z)
• Divergence is Uncertainty: A Closed-Form Posterior Covariance for Flow Matching [8.603039700922809]
  Flow matching has become a leading framework for generative modeling, but quantifying the uncertainty of its samples remains an open problem.<n>We show that none of these trade-offs is necessary.<n>Experiments on MNIST confirm that the resulting per-pixel uncertainty maps are semantically meaningful.
  arXiv  Detail & Related papers  (2026-05-01T04:25:00Z)
• Variance Is Not Importance: Structural Analysis of Transformer Compressibility Across Model Scales [0.0]
  spectral compression, block-level function replacement, rotation-based quantization, activation geometry, and adaptive early exit are studied.<n>We identify five structural properties relevant to compression.
  arXiv  Detail & Related papers  (2026-04-22T15:31:46Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation [64.92152574895111]
  We propose a simple Orthogonal Jacobian Regularization (OroJaR) to encourage deep generative model to learn disentangled representations. Our method is effective in disentangled and controllable image generation, and performs favorably against the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-17T15:01:46Z)
• Analytic Signal Phase in $N-D$ by Linear Symmetry Tensor--fingerprint modeling [69.35569554213679]
  We show that the Analytic Signal phase, and its gradient have a hitherto unstudied discontinuity in $2-D $ and higher dimensions. This shortcoming can result in severe artifacts whereas the problem does not exist in $1-D $ signals. We suggest the use of Linear Symmetry phase, relying on more than one set of Gabor filters, but with a negligible computational add-on.
  arXiv  Detail & Related papers  (2020-05-16T21:17:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.