When Do Task Vectors Interfere? Mapping the Validity Boundaries of Weight-Space Composition

• URL: http://arxiv.org/abs/2608.09490v1
• Date: Mon, 10 Aug 2026 11:58:04 GMT
• Title: When Do Task Vectors Interfere? Mapping the Validity Boundaries of Weight-Space Composition
• Abstract summary: Task arithmetic treats fine-tuning displacements as composable directions in weight space.<n>We separate parameter geometry from functional geometry and measure pairwise functional non-additivity over a two-dimensional task-vector surface.
• Score: 3.2114065314849323
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Task arithmetic treats fine-tuning displacements as composable directions in weight space, yet it remains unclear when parameter addition reflects predictable changes in model function. We separate parameter geometry from functional geometry and measure pairwise functional non-additivity over a two-dimensional task-vector surface, using a first-token predictive-distribution interaction ratio conditioned on an input distribution and evaluated with norm-matched controls, three training seeds, and response-only fine-tuning. On Qwen2.5-1.5B, code+safety is more non-additive than the matched code+math control on code and instruction prompts, but not on math prompts. In a prospectively specified six-task expansion, all eight high-versus-low comparisons of unseen task pairs have the predicted sign. The primary ordering further persists under full-parameter fine-tuning at 0.5B, Qwen2.5 LoRA scale tests up to 7B, and a Llama-3.1-8B cross-architecture audit. External validation exposes a sharper boundary: raw public code, instruction, and safety prompts preserve the continuous contrast, whereas an instruction-style wrapper collapses it on the identical public-code prompts, and EvalPlus pass@1 interactions do not robustly reproduce it. Weight-space composition therefore supports coarse, input- and format-conditioned functional statements across adaptation methods, scales, and one additional model family, not a universal merging-performance predictor.

Related papers

• SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation [51.46828526392219]
  Reasoning segmentation converts an implicit linguistic conclusion into a precise mask.<n>Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context.<n>We present SeGDeP, an explicit what-where interface.
  arXiv  Detail & Related papers  (2026-09-08T15:11:32Z)
• First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers [0.05729426778193398]
  We measure 8 properties with the same harness around a multitask LoRA operating point.<n>We find a shared one-direction validity window up to the tested scale $10-2$, but no universal radius for pairwise composition or update ordering.
  arXiv  Detail & Related papers  (2026-07-18T13:43:44Z)
• Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models [49.259951930558]
  Coding agents must integrate external tool returns into ongoing reasoning.<n>We exploit it through function-aware fill-in-the-middle mid-training.
  arXiv  Detail & Related papers  (2026-07-14T07:44:26Z)
• Amplitude-Only FFN Intervention for Tool-Structured LLM Inference Method: Gated Evaluation Protocol, and Cross-Model Empirical Results [10.526208975530054]
  We study inference-time feed-forward network (FFN) intervention for improving structured outputs without retraining model weights.<n>Amplitude Gating (AG) is a non-destructive alternative that preserves pretrained FFN weight directions and modulates only activation magnitudes during generation.<n>These results identify tool-structured inference as the most credible first target for safe FFN-level inference optimization.
  arXiv  Detail & Related papers  (2026-07-13T07:28:55Z)
• Closed-Loop Control with Rule-Aligned Small Language Models and Multi-Agent Self-Correction [17.527470001317685]
  This work investigates whether a compact Small Language Model (SLM) can be retrained for control reasoning and embedded in a validator-guided correction loop.<n>In randomized thermal-control simulations (30 experiments with 500 steps each), the framework achieves 91.5% average action-alignment accuracy.<n>Under symbolic re-mapping, it maintains a 95% in-range rate, indicating robust physical regulation despite reduced token-level agreement.
  arXiv  Detail & Related papers  (2026-06-24T13:49:01Z)
• PACT: Preserving Anchored Cores in Task-vectors for Model Merging [68.52455853496585]
  Model merging aims to combine multiple task-specific fine-tuned models into a single multi-task model.<n>Most existing model merging approaches follow the Task Arithmetic paradigm.<n>We propose PACT, which preserves the anchored task-specific cores (i.e., LBW dimensions) within task vectors by aligning their complements with the subspace of the pre-trained weights.
  arXiv  Detail & Related papers  (2026-06-17T02:48:35Z)
• Structural Rigidity and the 57-Token Predictive Window: A Physical Framework for Inference-Layer Governability in Large Language Models [0.0]
  Current AI safety relies on behavioral monitoring and post-training alignment.<n> empirical measurement shows these approaches produce no detectable pre-commitment signal.<n>We present an energy-based governance framework connecting transformer inference dynamics to constraint-satisfaction models.
  arXiv  Detail & Related papers  (2026-04-04T00:08:17Z)
• Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation [55.938648534942665]
  Test-time policy adaptation for multi-turn interactions (T2PAM) is essential for aligning Large Language Models (LLMs) with dynamic user needs during inference time.<n>We propose ROSA2, a framework that reformulates interaction as a joint optimization problem over the heterogeneous space of Words and Weights.
  arXiv  Detail & Related papers  (2026-03-02T02:16:20Z)
• Sparse Semantic Dimension as a Generalization Certificate for LLMs [53.681678236115836]
  We introduce the Sparse Semantic Dimension (SSD), a complexity measure derived from the active feature vocabulary of a Sparse Autoencoder (SAE) trained on the model's layers.<n>We validate this framework on GPT-2 Small and Gemma-2B, demonstrating that our bound provides non-vacuous certificates at realistic sample sizes.
  arXiv  Detail & Related papers  (2026-02-11T21:45:18Z)
• Semi-DETR: Semi-Supervised Object Detection with Detection Transformers [105.45018934087076]
  We analyze the DETR-based framework on semi-supervised object detection (SSOD) We present Semi-DETR, the first transformer-based end-to-end semi-supervised object detector. Our method outperforms all state-of-the-art methods by clear margins.
  arXiv  Detail & Related papers  (2023-07-16T16:32:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.