Spectral Rewiring for Exploration, Purification, and Model Merging

• URL: http://arxiv.org/abs/2607.03065v1
• Date: Fri, 03 Jul 2026 07:57:31 GMT
• Title: Spectral Rewiring for Exploration, Purification, and Model Merging
• Abstract summary: Subspace-Aligned Rewiring (SAR) is a post-hoc editing method that retains reasoning gains while removing components.<n>We show that SAR preserves over 99% of post-training performance and improves high-k exploration in mathematical reasoning.<n>It also purifies mixed-domain training updates by releasing suppressed coding capability while maintaining math reasoning and instruction following.
• Score: 40.70734593668918
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning has become a standard post-training recipe for large language models, but dense full-parameter updates create two deployment-relevant bottlenecks: suppressed reasoning performance, often reflected by premature saturation of test-time scaling, and interference when consolidating multiple capabilities through multi-domain training or model merging. We show that the reasoning-effective component of these updates is largely concentrated in the base model's spectral space, motivating Subspace-Aligned Rewiring (SAR), a post-hoc editing method that retains this spectral core while removing orthogonal components. SAR therefore preserves reasoning gains and filters residual update directions that suppress performance or amplify cross-domain interference. Across several model families and scales, SAR extracts compact reasoning cores using as little as approximately 0.58% of total parameters: it preserves over 99% of post-training performance and improves high-k exploration in mathematical reasoning, and generalizes to agentic coding by improving six of seven open benchmarks on an in-house model. SAR also purifies mixed-domain training updates by releasing suppressed coding capability while maintaining math reasoning and instruction following. It further enables model merging across experts, yielding cross-domain generalization that surpasses previous merging baselines and even the best single-domain experts. Overall, SAR shows that extracting reasoning-effective updates from parameter geometry can serve as a training-free mechanism to improve reasoning and multi-domain performance.

Related papers

• GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution [5.770893169582546]
  This work reframes attribution as subset-level counterfactual utility prediction.<n>We introduce GRASP, an interaction-aware surrogate.<n>It decisively outperforms existing scalable baselines.
  arXiv  Detail & Related papers  (2026-06-05T04:17:50Z)
• Inference-Time Scaling in Diffusion Models through Iterative Partial Refinement [22.349959761835468]
  Inference-time scaling has emerged as a major approach for improving reasoning capabilities.<n>We propose Iterative Partial Refinement (IPR), an inference-time scaling method for sequential diffusion.<n>IPR produces more globally consistent samples without external verification.
  arXiv  Detail & Related papers  (2026-05-19T03:53:50Z)
• FRACTAL: SSM with Fractional Recurrent Architecture for Computational Temporal Analysis of Long Sequences [13.320173396455226]
  This paper proposes a Fractional Recurrent Architecture for Computational Temporal Analysis of Long sequences (FRACTAL)<n>By deriving projection operators with analytically characterized spectral properties and a tunable index, the proposed method sensitivity to recent signal perturbations while preserving the spectral structure that encodes fractional scale-invariant memory dynamics.<n>FRACTAL achieves an average score of 87.11% on the Long Range Arena benchmark, including 61.85% on the ListOps task, outperforming the S5 model.
  arXiv  Detail & Related papers  (2026-05-09T09:38:44Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models [51.45122910254346]
  Reinforcement learning (RL)-based post-training often improves the reasoning performance of large language models beyond the training domain.<n>Supervised fine-tuning (SFT) frequently leads to general forgetting capabilities.<n>We present a feature-level mechanistic analysis methodology to probe RL generalization using a controlled experimental setup.
  arXiv  Detail & Related papers  (2026-04-27T21:22:34Z)
• Recursive Concept Evolution for Compositional Reasoning in Large Language Models [0.0]
  Large language models achieve strong performance on many complex reasoning tasks, yet their accuracy degrades sharply on benchmarks that require compositional reasoning.<n>We propose Recursive Concept Evolution (RCE), a framework that enables pretrained language models to modify their internal representation geometry during inference.<n>RCE yields 12-18 point gains on ARC-AGI-2, 8-14 point improvements on GPQA and BBH, and consistent reductions in depth-induced error on MATH and HLE.
  arXiv  Detail & Related papers  (2026-02-17T17:01:42Z)
• Any Image Restoration via Efficient Spatial-Frequency Degradation Adaptation [158.37640586809187]
  Restoring any degraded image efficiently via just one model has become increasingly significant.<n>Our approach, termed AnyIR, takes a unified path that leverages inherent similarity across various degradations.<n>To fuse the degradation awareness and the contextualized attention, a spatial-frequency parallel fusion strategy is proposed.
  arXiv  Detail & Related papers  (2025-04-19T09:54:46Z)
• SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learning [6.262268096839562]
  Domain generalization aims to adapt a model using one or multiple source domains to ensure robust performance in unseen target domains.<n>Existing PEFT methods struggle to strike a balance between preserving generalizable components of the pre-trained model and learning task-specific features.<n>We introduce Singular Value De Minor Components Adaptation (SoMA), an approach that selectively tunes minor singular components while keeping the residual parts frozen.
  arXiv  Detail & Related papers  (2024-12-05T11:17:57Z)
• Consensus-Adaptive RANSAC [104.87576373187426]
  We propose a new RANSAC framework that learns to explore the parameter space by considering the residuals seen so far via a novel attention layer. The attention mechanism operates on a batch of point-to-model residuals, and updates a per-point estimation state to take into account the consensus found through a lightweight one-step transformer.
  arXiv  Detail & Related papers  (2023-07-26T08:25:46Z)
• Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression [53.15502562048627]
  Recent work has built the connection between self-supervised learning and the approximation of the top eigenspace of a graph Laplacian operator. This work delves into a statistical analysis of augmentation-based pretraining.
  arXiv  Detail & Related papers  (2023-06-01T15:18:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.