Leveraging Instruction Tuning and Merging for Reasoning Model Adaptation

• URL: http://arxiv.org/abs/2607.14895v1
• Date: Thu, 16 Jul 2026 12:11:25 GMT
• Title: Leveraging Instruction Tuning and Merging for Reasoning Model Adaptation
• Abstract summary: Reasoning language models (RLMs) have demonstrated impressive performance in domains such as mathematics and coding.<n>For both verifiable and unverifiable domains, large amounts of unused supervised fine-tuning data with human-written solutions exist.<n>In this work, we show that these data can be used efficiently to further improve RLM performance.
• Score: 7.414725419251954
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning language models (RLMs) have demonstrated impressive performance in domains such as mathematics and coding. These domains permit reliable verification of model outputs, which is important for enabling the reinforcement learning that drives RLM performance gains. However, training RLMs on domains that lack reliable verifiers remains challenging. Meanwhile, for both verifiable and unverifiable domains, large amounts of unused supervised fine-tuning data with human-written solutions exist. In this work, we show that these data can be used efficiently to further improve RLM performance. For this, we first use classic instruction tuning, supervised fine-tuning without reasoning traces, on the RLM. Next, we merge our instruction-tuned model with the original reasoning model, recovering its reasoning behavior on the target domain. Our extensive evaluation demonstrates that our technique improves RLM performance in both verifiable and hard-to-verify domains, including coding and text summarization, while preserving RLM capabilities across other domains. Importantly, our method is highly cost-effective, enabling such improvements for less than USD $3.

Related papers

• CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents [66.9575676266637]
  Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments.<n>In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution.<n>We develop CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization.
  arXiv  Detail & Related papers  (2026-08-31T01:59:51Z)
• Constructing Synthetic Instruction Datasets for Improving Reasoning in Domain-Specific LLMs: A Case Study in the Japanese Financial Domain [0.1529342790344802]
  This study proposes a general method for constructing high-quality synthetic instruction data for any domain.<n>We constructed a large-scale instruction dataset totaling approximately 9.5 billion tokens with Chain-of-Thought reasoning traces.
  arXiv  Detail & Related papers  (2026-03-02T01:21:54Z)
• Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling [21.45871501724415]
  We propose a data-driven approach to automatically construct highly granular reasoning model error.<n>Rubrics can be used to build stronger LLM-as-judge reward functions.<n>This extension opens the door for teaching models to solve complex technical problems without a full dataset of gold labels.
  arXiv  Detail & Related papers  (2026-02-06T15:51:52Z)
• WeaveRec: An LLM-Based Cross-Domain Sequential Recommendation Framework with Model Merging [24.949880939628386]
  We introduce WeaveRec, which cross-trains multiple LoRA modules with source and target domain data in a weaving fashion.<n>We provide a theoretical guarantee that WeaveRec can reduce the upper bound of the expected error in the target domain.
  arXiv  Detail & Related papers  (2025-10-30T14:37:15Z)
• How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices [81.85465545346266]
  Diffusion language models (DLMs) have emerged as a promising alternative to the long-dominant autoregressive (AR) paradigm.<n>Yet, current open-source DLMs often underperform their AR counterparts in speed, limiting their real-world utility.<n>This work presents a systematic study of DLM efficiency, identifying key issues in prior evaluation methods.
  arXiv  Detail & Related papers  (2025-10-21T10:00:32Z)
• KARE-RAG: Knowledge-Aware Refinement and Enhancement for RAG [63.82127103851471]
  Retrieval-Augmented Generation (RAG) enables large language models to access broader knowledge sources.<n>We demonstrate that enhancing generative models' capacity to process noisy content is equally critical for robust performance.<n>We present KARE-RAG, which improves knowledge utilization through three key innovations.
  arXiv  Detail & Related papers  (2025-06-03T06:31:17Z)
• General-Reasoner: Advancing LLM Reasoning Across All Domains [64.70599911897595]
  Reinforcement learning (RL) has recently demonstrated strong potential in enhancing the reasoning capabilities of large language models (LLMs)<n>We propose General-Reasoner, a novel training paradigm designed to enhance LLM reasoning capabilities across diverse domains.<n>We train a series of models and evaluate them on a wide range of datasets covering wide domains like physics, chemistry, finance, electronics etc.
  arXiv  Detail & Related papers  (2025-05-20T17:41:33Z)
• R-PRM: Reasoning-Driven Process Reward Modeling [53.06844294668382]
  Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step.<n>Existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy.<n>We propose Reasoning-Driven Process Reward Modeling (R-PRM)<n>R-PRM generates seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities.
  arXiv  Detail & Related papers  (2025-03-27T09:23:08Z)
• Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning [65.23593936798662]
  We show that fine-tuning with LLM-generated data improves target task performance and reduces non-target task degradation.<n>This is the first work to provide an empirical explanation based on token perplexity reduction to mitigate catastrophic forgetting in LLMs after fine-tuning.
  arXiv  Detail & Related papers  (2025-01-24T08:18:56Z)
• RE-AdaptIR: Improving Information Retrieval through Reverse Engineered Adaptation [37.969478059005574]
  Large language models (LLMs) fine-tuned for text-retrieval have demonstrated state-of-the-art results across several information retrieval benchmarks. We explore the effectiveness of extending reverse engineered adaptation to the context of information retrieval.
  arXiv  Detail & Related papers  (2024-06-20T22:28:11Z)
• Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Models [0.8399688944263842]
  Large Language Models (LLMs) have the capability to understand and generate human-like text from input queries. This study extends this concept to the integration of LLMs within Retrieval-Augmented Generation (RAG) pipelines. We evaluate the impact of fine-tuning on the LLMs' capacity for data extraction and contextual understanding.
  arXiv  Detail & Related papers  (2024-06-17T04:35:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.