Online Data Selection Is Implicit Alignment
- URL: http://arxiv.org/abs/2607.07023v1
- Date: Wed, 08 Jul 2026 05:44:33 GMT
- Title: Online Data Selection Is Implicit Alignment
- Abstract summary: We study online data selection as an implicit alignment mechanism.<n>We show that selectors are statistically indistinguishable in task accuracy diverge sharply in refusal rate, verbosity, and sycophancy.<n>This view predicts that high-scoring data can systematically favor longer, more assertive, more compliant, or more refusal-prone behaviors depending on how the online score is defined.
- Score: 1.9054869116450563
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Supervised fine-tuning (SFT) is often treated as a capability-adaptation step, while alignment is attributed to later preference optimization or reinforcement learning. This separation is incomplete: when examples are scored and kept online during fine-tuning, the choice of which data to train on already changes the model's behavioral preferences. We study online data selection as an implicit alignment mechanism. Given the same base model, optimizer, and selected-token budget, we compare random, loss-based, quality-based, and diversity-based online selectors and measure the behavioral drift they induce without any preference optimization. The proposed evaluation tracks helpfulness, refusal rate, verbosity, truthfulness, sycophancy, calibration, and jailbreak robustness, together with diagnostics for which behavioral modes are over-represented in the selected data. We formalize online selection as a reweighted SFT objective whose weights define an implicit preference over response styles and safety postures, so that an online scorer plays the role usually assigned to a reward model. This view predicts that high-scoring data can systematically favor longer, more assertive, more compliant, or more refusal-prone behaviors depending on how the online score is defined. Empirically, selectors that are statistically indistinguishable in task accuracy diverge sharply in refusal rate, verbosity, and sycophancy, and we show that the direction of the shift is predictable from the attribute mixture of the selected data. We introduce Alignment Drift Auditing (ADA), a controlled protocol for quantifying selection-induced behavioral movement, and Alignment-Aware Selection (AAS), a diagnostic online selector that retains data efficiency while constraining drift along safety and style axes.
Related papers
- Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training [36.289092974454796]
We show that selected-data training induces an implicit regularization effect by the instantaneous selection ratio.<n>This reveals a key trade-off: lower amplify selection-induced regularization, whereas higher ratios preserve data coverage.<n> PODS is compatible with existing static and dynamic selection methods and broadly applicable across training paradigms.
arXiv Detail & Related papers (2026-05-14T12:37:11Z) - Two-Stage Optimizer-Aware Online Data Selection for Large Language Models [49.576993784867035]
We propose a principled-aware framework for gradient-based online data selection and reweighting in large language model fine-tuning.<n>Our key idea is to view online selection not as static sample ranking, but as shaping the next target-oriented update under the state.<n> Experiments show that our method consistently improves convergence and downstream performance over existing online data selection baselines under the same data budget.
arXiv Detail & Related papers (2026-03-08T21:46:16Z) - A Unified Understanding of Offline Data Selection and Online Self-refining Generation for Post-training LLMs [55.931369468485464]
We tackle offline data selection and online self-refining generations through an optimization perspective.<n>For the first time, we theoretically demonstrate the effectiveness of the bilevel data selection framework.
arXiv Detail & Related papers (2025-11-26T04:48:33Z) - ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning [51.133569963553576]
ssToken is a Self-modulated and Semantic-aware Token Selection approach.<n>We show that both self-modulated selection and semantic-aware selection alone outperform full-data fine-tuning.
arXiv Detail & Related papers (2025-10-21T03:21:04Z) - Online Conformal Selection with Accept-to-Reject Changes [14.619101485265322]
Online Conformal Selection with Accept-to-Reject Changes (dubbed OCS-ARC) is proposed.<n>It incorporates online Benjamini-Hochberg procedure into the candidate selection process.<n>We provide theoretical guarantees that OCS-ARC controls the false discovery rate (FDR) at or below the nominal level at any timestep.
arXiv Detail & Related papers (2025-08-19T13:58:38Z) - Online selective conformal inference: adaptive scores, convergence rate and optimality [4.7198252163006345]
We introduce an extended version of the point-prediction algorithm, called OnlineSCI, allowing the user to select times where such an inference should be made.<n>OnlineSCI encompasses several prominent online selective tasks, such as building prediction intervals for extreme outcomes, classification with abstention, and online testing.<n>We show that the adaptive versions of OnlineSCI can convergence to an optimal solution and provide an explicit convergence rate in each of the aforementioned application cases.
arXiv Detail & Related papers (2025-08-14T04:36:14Z) - CAP: A General Algorithm for Online Selective Conformal Prediction with FCR Control [4.137346786534721]
It is important to control the real-time false coverage-statement rate (FCR) which measures the overall miscoverage level.<n>We develop a general framework named CAP that performs an adaptive pick rule on historical data to construct a calibration set.<n>We prove that CAP can achieve an exact selection-conditional coverage guarantee in the finite-sample and distribution-free regimes.
arXiv Detail & Related papers (2024-03-12T15:07:20Z) - Data-Driven Offline Decision-Making via Invariant Representation
Learning [97.49309949598505]
offline data-driven decision-making involves synthesizing optimized decisions with no active interaction.
A key challenge is distributional shift: when we optimize with respect to the input into a model trained from offline data, it is easy to produce an out-of-distribution (OOD) input that appears erroneously good.
In this paper, we formulate offline data-driven decision-making as domain adaptation, where the goal is to make accurate predictions for the value of optimized decisions.
arXiv Detail & Related papers (2022-11-21T11:01:37Z) - Online Coreset Selection for Rehearsal-based Continual Learning [65.85595842458882]
In continual learning, we store a subset of training examples (coreset) to be replayed later to alleviate catastrophic forgetting.
We propose Online Coreset Selection (OCS), a simple yet effective method that selects the most representative and informative coreset at each iteration.
Our proposed method maximizes the model's adaptation to a target dataset while selecting high-affinity samples to past tasks, which directly inhibits catastrophic forgetting.
arXiv Detail & Related papers (2021-06-02T11:39:25Z) - Tracking Performance of Online Stochastic Learners [57.14673504239551]
Online algorithms are popular in large-scale learning settings due to their ability to compute updates on the fly, without the need to store and process data in large batches.
When a constant step-size is used, these algorithms also have the ability to adapt to drifts in problem parameters, such as data or model properties, and track the optimal solution with reasonable accuracy.
We establish a link between steady-state performance derived under stationarity assumptions and the tracking performance of online learners under random walk models.
arXiv Detail & Related papers (2020-04-04T14:16:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.