Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

• URL: http://arxiv.org/abs/2604.13538v1
• Date: Wed, 15 Apr 2026 06:37:26 GMT
• Title: Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
• Abstract summary: We propose a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation.<n>Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses.
• Score: 17.127903764198084
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures.

Related papers

• Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoders [20.530146256745414]
  EPIC is a novel embedding-based in-context prompt training strategy for large language models.<n>It generates high-quality embeddings while reducing computational burden during both training and inference.
  arXiv  Detail & Related papers  (2026-05-02T10:31:01Z)
• FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale [56.03294218908334]
  Large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective.<n>To make the model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses.<n>We propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs.
  arXiv  Detail & Related papers  (2026-01-29T18:58:47Z)
• Enhancing Training Data Attribution with Representational Optimization [57.61977909113113]
  Training data attribution methods aim to measure how training data impacts a model's predictions.<n>We propose AirRep, a representation-based approach that closes this gap by learning task-specific and model-aligned representations explicitly for TDA.<n>AirRep introduces two key innovations: a trainable encoder tuned for attribution quality, and an attention-based pooling mechanism that enables accurate estimation of group-wise influence.
  arXiv  Detail & Related papers  (2025-05-24T05:17:53Z)
• Aligning Instruction Tuning with Pre-training [61.50161961371844]
  We propose Aligning Instruction Tuning with Pre-training (AITP) to align instruction tuning with pre-training distributions.<n>We show consistent performance improvements with AITP on three fully open large language models (LLMs) across eight benchmarks.
  arXiv  Detail & Related papers  (2025-01-16T08:27:40Z)
• Revealing the Inherent Instructability of Pre-Trained Language Models [6.293897980482996]
  We show that Response Tuning (RT) removes the instruction and its corresponding mapping to the response from instruction tuning.<n>Our experiments demonstrate that RT, trained only on responses, can effectively respond to a wide range of instructions akin to their instruction-tuned counterparts.<n>These findings support our hypothesis, highlighting the inherent capabilities of pre-trained LLMs.
  arXiv  Detail & Related papers  (2024-10-03T13:15:19Z)
• DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning [75.68193159293425]
  In-context learning (ICL) allows transformer-based language models to learn a specific task with a few "task demonstrations" without updating their parameters.<n>We propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL.<n>We experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.
  arXiv  Detail & Related papers  (2024-05-22T15:52:52Z)
• Mosaic-IT: Cost-Free Compositional Data Synthesis for Instruction Tuning [30.82220015525281]
  Mosaic Instruction Tuning (Mosaic-IT) is a human/model-free compositional data synthesis method.<n>Our evaluations demonstrate a superior performance and training efficiency of Mosaic-IT.
  arXiv  Detail & Related papers  (2024-05-22T04:08:20Z)
• From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning [63.63840740526497]
  We investigate how instruction tuning adjusts pre-trained models with a focus on intrinsic changes. The impact of instruction tuning is then studied by comparing the explanations derived from the pre-trained and instruction-tuned models. Our findings reveal three significant impacts of instruction tuning.
  arXiv  Detail & Related papers  (2023-09-30T21:16:05Z)
• Instruction Position Matters in Sequence Generation with Large Language Models [67.87516654892343]
  Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization. We propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences.
  arXiv  Detail & Related papers  (2023-08-23T12:36:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.