Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectors

• URL: http://arxiv.org/abs/2605.00610v1
• Date: Fri, 01 May 2026 12:20:44 GMT
• Title: Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectors
• Abstract summary: We analyze SFT and RLVR through the lens of task vectors.<n>We propose Decoupled Test-time Synthesis (DoTS) to allow SFT and RLVR checkpoints to be trained independently.
• Score: 26.233592394784868
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: SFT and RLVR represent two fundamental yet distinct paradigms for LLM post-training, each excelling in distinct dimensions. SFT expands knowledge breadth while RLVR enhances reasoning depth. Yet integrating these complementary strengths remains a formidable challenge. Sequential training can cause catastrophic forgetting, and joint optimization often suffers from severe gradient conflicts. We analyze SFT and RLVR through the lens of task vectors and reveal three structural properties behind these failures: a 30* magnitude disparity, 45* sign interference, and heterogeneous module-wise update distributions. These findings show SFT and RLVR are difficult to integrate directly, but they also suggest that the two paradigms modify partly complementary components of the model. Motivated by these observations, we propose Decoupled Test-time Synthesis (DoTS), a post-hoc framework allows SFT and RLVR checkpoints to be trained independently and synthesizes their capabilities only at inference time via task vector arithmetic, without updating model parameters. To reduce interference, DOTS applies selective sparsification with norm-preserving rescaling. It then uses Bayesian optimization on a small set of unlabeled queries to search for combination coefficients on the Pareto frontier of consistency and perplexity. Empirically, \ours matches or exceeds the performance of training-based SFT--RLVR integration methods across multiple mathematical reasoning benchmarks, incurring only $\sim$3\% of the computational cost. When applied to stronger post-trained checkpoints, DOTS surpasses SOTA models and generalizes to out-of-domain benchmarks without re-tuning. Code is available at https://github.com/chaohaoyuan/DoTS.

Related papers

• Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging [35.66009296523317]
  Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful post-training paradigm that surpasses Supervised Fine-Tuning (SFT)<n>Recent studies reveal that RLVR induces highly sparse and off-principal parameter updates compared to SFT.<n>We propose Sensitivity-aware Resolving Merging (SAR-Merging), a merging recipe tailored for the unique structure of RLVR parameter spaces.
  arXiv  Detail & Related papers  (2026-06-16T22:22:40Z)
• Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance [117.2119290254454]
  We propose FEST, a demonstration-guided RLVR algorithm.<n>It attains compelling results with only 128 demonstrations randomly selected from an SFT dataset.<n>We find that three components are vital for the success: supervised signal, on-policy signal, and decaying weights on the few-shot SFT dataset.
  arXiv  Detail & Related papers  (2026-05-14T16:12:30Z)
• Relatron: Automating Relational Machine Learning over Relational Databases [50.94254514286021]
  We present a study that unifies RDL and DFS in a shared design space and conducts architecture-centric searches across diverse RDB tasks.<n>Our analysis yields three key findings: (1) RDL does not consistently outperform DFS, with performance being highly task-dependent; (2) no single architecture dominates across tasks, underscoring the need for task-aware model selection; and accuracy is an unreliable guide for choice architecture.
  arXiv  Detail & Related papers  (2026-02-26T02:45:22Z)
• The Path Not Taken: RLVR Provably Learns Off the Principals [85.41043469428365]
  We show that sparsity is a surface artifact of a model-conditioned optimization bias.<n>We mechanistically explain these dynamics with a Three-Gate Theory.<n>We provide a parameter-level characterization of RLVR's learning dynamics.
  arXiv  Detail & Related papers  (2025-11-11T18:49:45Z)
• ViSurf: Visual Supervised-and-Reinforcement Fine-Tuning for Large Vision-and-Language Models [52.819394001139955]
  Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) are typical post-training paradigms for Large Vision-and-Language Models (LVLMs)<n>We propose ViSurf, a unified post-training paradigm that integrates the strengths of both SFT and RLVR within a single stage.<n>The core of ViSurf involves injecting ground-truth labels into the RLVR rollouts, thereby providing simultaneous external supervision and internal reinforcement.
  arXiv  Detail & Related papers  (2025-10-12T13:42:55Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• Harnessing Optimization Dynamics for Curvature-Informed Model Merging [17.42364575754576]
  In supervised fine-tuning, multiple capability-based SFT checkpoints must be consolidated into a single model.<n>We introduce Optimization Trajectory Aware (OTA) Merging and Fast Fisher Grafting (FFG)<n>OTA+FFG improves merged-model quality over strong weight-space baselines, reduces negative transfer, and remains robust across sparsity levels.
  arXiv  Detail & Related papers  (2025-09-14T08:59:53Z)
• Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum [78.27945336558987]
  Decentralized server (DFL) eliminates reliance on client-client architecture.<n>Non-smooth regularization is often incorporated into machine learning tasks.<n>We propose a novel novel DNCFL algorithm to solve these problems.
  arXiv  Detail & Related papers  (2025-04-17T08:32:25Z)
• Sequential Federated Learning in Hierarchical Architecture on Non-IID Datasets [25.010661914466354]
  In a real federated learning (FL) system, communication overhead for passing model parameters between the clients and the parameter (PS) is often a bottleneck. We propose sequential FL (SFL) HFL for the first time, which removes the central PS and enables the model to be completed only through passing data between two adjacent ESs for each server.
  arXiv  Detail & Related papers  (2024-08-19T07:43:35Z)
• Unlock the Correlation between Supervised Fine-Tuning and Reinforcement Learning in Training Code Large Language Models [12.656574142412484]
  We make an attempt to understand the correlation between supervised fine-tuning and reinforcement learning.<n>We find that both atomic and synthetic functions are indispensable for SFT's generalization.
  arXiv  Detail & Related papers  (2024-06-14T03:39:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.