A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design

• URL: http://arxiv.org/abs/2606.11189v1
• Date: Tue, 09 Jun 2026 17:59:54 GMT
• Title: A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design
• Abstract summary: Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory.<n>Strictly fitting toward this one-hot target may be suboptimal, especially when the pretrained model encodes a rich knowledge prior.<n>In this work, we reinterpret SFT as target distribution design: instead of studying only the loss objective, we analyze the token-level target that the loss drives the model to match.
• Score: 47.42907478044802
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory. However, an observed token can be non-unique, noisy, or misaligned with the model prior. Strictly fitting toward this one-hot target may be suboptimal, especially when the pretrained model encodes a rich knowledge prior. In this work, we reinterpret SFT as target distribution design: instead of studying only the loss objective, we analyze the token-level target that the loss drives the model to match. We introduce the Q-target framework, which decomposes SFT supervision into two explicit choices: (1) how strongly to rely on the observed token, and (2) how to allocate the remaining probability mass over alternatives. This perspective unifies many existing SFT variants as implicit choices of the target distribution Q. Building on this view, we propose Target-SFT which constructs the training objective directly from the desired target distribution. This method consistently outperforms across the ten reasoning dataset-model settings evaluated, showing the effectiveness of this target-based approach. Overall, our formulation reveals a more fundamental design principle for SFT training and opens a broader search space for SFT objectives.

Related papers

• From Two Passes to One: Compact and Efficient Target-Stance Extraction [0.0]
  We present a one-pass, joint architecture that predicts both in a single forward pass.<n>Standard target-scrubbing practices artificially suppress target prediction accuracy.<n> Retaining explicit target mentions improves F1 by at least 6 points across both target classification and target generation settings.
  arXiv  Detail & Related papers  (2026-09-05T14:07:43Z)
• Action-BED: Task-Driven Bayesian Experimental Design with Singly Intractable Objectives [12.963546988204875]
  We show that an expected future loss (EFL) on downstream actions can provide a simple and naturally task-driven framework for design policies.<n>We then show that all such EFLs can be rearranged into singly intractable objectives that can be jointly optimised with respect to both the design policy and a downstream action policy.
  arXiv  Detail & Related papers  (2026-06-22T17:48:07Z)
• Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance [92.72779885657373]
  We propose a framework that grounds model selection in the internal functional dynamics of the visual encoder.<n>Our approach represents each task via layer wise conductance and derives a target-conditioned block importance distribution through entropy regularized alignment.<n>Building on this, we introduce Directional Conductance Divergence (DCD), an asymmetric metric that quantifies how effectively a source task covers the target's salient functional blocks.
  arXiv  Detail & Related papers  (2026-02-01T17:29:43Z)
• Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity [13.211627219720796]
  Reinforcement Learning (RL) has become the de facto standard for tuning LLMs to solve tasks involving reasoning.<n>We argue that RL implicitly optimize the "mode-seeking" or "zero-forcing" Reverse KL to a target distribution causing the model to concentrate mass on certain high-probability regions of the target while others.<n>In this work, we instead begin from an explicit target distribution, obtained by filtering out incorrect answers while neglecting the relative probabilities of correct ones.
  arXiv  Detail & Related papers  (2025-12-05T18:56:40Z)
• Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum [88.90314335542281]
  We study a family of probability-based objectives and characterize their effectiveness under different conditions.<n>We uncover a critical dimension that governs objective behavior: the model-capability continuum.<n>Our theoretical analysis further elucidates how objectives trade places across the continuum, providing a principled foundation for adapting objectives to model capability.
  arXiv  Detail & Related papers  (2025-10-01T05:17:47Z)
• When few labeled target data suffice: a theory of semi-supervised domain adaptation via fine-tuning from multiple adaptive starts [5.839411310096219]
  Semi-supervised domain adaptation (SSDA) aims to achieve high predictive performance in the target domain with limited labeled target data.<n>We develop a theoretical framework based on structural causal models (SCMs) which allows us to analyze and quantify the performance of SSDA methods when labeled target data is limited.<n>We propose the Multi Adaptive-Start Fine-Tuning (MASFT) algorithm, which fine-tunes UDA models from multiple starting points and selects the best-performing one.
  arXiv  Detail & Related papers  (2025-07-19T15:18:28Z)
• Aligning Latent Spaces with Flow Priors [72.24305287508474]
  This paper presents a novel framework for aligning learnable latent spaces to arbitrary target distributions by leveraging flow-based generative models as priors.<n> Notably, the proposed method eliminates computationally expensive likelihood evaluations and avoids ODE solving during optimization.
  arXiv  Detail & Related papers  (2025-06-05T16:59:53Z)
• Self-training through Classifier Disagreement for Cross-Domain Opinion Target Extraction [62.41511766918932]
  Opinion target extraction (OTE) or aspect extraction (AE) is a fundamental task in opinion mining. Recent work focus on cross-domain OTE, which is typically encountered in real-world scenarios. We propose a new SSL approach that opts for selecting target samples whose model output from a domain-specific teacher and student network disagrees on the unlabelled target data.
  arXiv  Detail & Related papers  (2023-02-28T16:31:17Z)
• TarGF: Learning Target Gradient Field for Object Rearrangement [8.49306925839127]
  We focus on a more practical setting in object rearrangement, i.e., rearranging objects from shuffled layouts to a normative target distribution. It is hard to describe the target distribution (goal specification) for reward engineering or collect expert trajectories as demonstrations. We employ the score-matching objective to train a Target Gradient Field (TarGF), indicating a direction on each object to increase the likelihood of the target distribution.
  arXiv  Detail & Related papers  (2022-09-02T07:20:34Z)
• Generative multitask learning mitigates target-causing confounding [61.21582323566118]
  We propose a simple and scalable approach to causal representation learning for multitask learning. The improvement comes from mitigating unobserved confounders that cause the targets, but not the input. Our results on the Attributes of People and Taskonomy datasets reflect the conceptual improvement in robustness to prior probability shift.
  arXiv  Detail & Related papers  (2022-02-08T20:42:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.