Two-Stage Optimizer-Aware Online Data Selection for Large Language Models

• URL: http://arxiv.org/abs/2604.00001v1
• Date: Sun, 08 Mar 2026 21:46:16 GMT
• Title: Two-Stage Optimizer-Aware Online Data Selection for Large Language Models
• Abstract summary: We propose a principled-aware framework for gradient-based online data selection and reweighting in large language model fine-tuning.<n>Our key idea is to view online selection not as static sample ranking, but as shaping the next target-oriented update under the state.<n> Experiments show that our method consistently improves convergence and downstream performance over existing online data selection baselines under the same data budget.
• Score: 49.576993784867035
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Gradient-based data selection offers a principled framework for estimating sample utility in large language model (LLM) fine-tuning, but existing methods are mostly designed for offline settings. They are therefore less suited to online fine-tuning, where data arrives sequentially, sample utility is step-dependent, and the effective update geometry is shaped by adaptive optimizers. We propose an optimizer-aware framework for gradient-based online data selection and reweighting in LLM fine-tuning. Our key idea is to view online selection not as static sample ranking, but as shaping the next target-oriented update under the optimizer state. We formulate this as an optimizer-aware update-matching problem, establish its connection to second-order target utility, and show why subset-level construction must account for interactions and redundancy among selected samples. Based on this view, we develop a two-stage Filter-then-Weight algorithm that first filters geometrically useful candidates and then optimizes their coefficients. To make the framework practical for LLMs, we introduce a factorized outer-product gradient representation and optimized matrix computations for long-context data. Experiments show that our method consistently improves convergence and downstream performance over existing online data selection baselines under the same data budget.

Related papers

• Rethinking Learnability in Offline Data-driven Optimization [20.146021332129703]
  Black-Box Optimization (BBO) faces efficiency challenges as real-world problems grow increasingly complex.<n>Data-driven optimization has been the most popular paradigm to improve the efficiency of BBO, by learning from data.<n>Many offline optimization methods have been proposed, but a fundamental question remains unanswered: what learnability is sufficient for offline optimization?
  arXiv  Detail & Related papers  (2026-09-01T16:22:28Z)
• A Unified Understanding of Offline Data Selection and Online Self-refining Generation for Post-training LLMs [55.931369468485464]
  We tackle offline data selection and online self-refining generations through an optimization perspective.<n>For the first time, we theoretically demonstrate the effectiveness of the bilevel data selection framework.
  arXiv  Detail & Related papers  (2025-11-26T04:48:33Z)
• Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization [32.84452172765487]
  Preference optimization is crucial for aligning large language models with human values and intentions.<n>Existing methods attempt to reduce this gap using statics or decoupled online sampling strategies.<n>We propose Meta-Weighted Adaptive Preference Optimization (MetaAPO), a novel framework that dynamically couples data generation with model training.
  arXiv  Detail & Related papers  (2025-09-27T15:38:24Z)
• Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization [37.54165341391688]
  We introduce a novel problem: Sample Scheduling for DPO.<n>We propose SamS, an efficient and effective algorithm that adaptively selects samples in each training batch.<n>This work points to a promising new direction for improving LLM alignment through batch-wise sample selection.
  arXiv  Detail & Related papers  (2025-06-08T10:26:09Z)
• Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization [14.50339880957898]
  We aim to improve the preference optimization pipeline by taking a closer look at preference data generation and training regularization techniques. For preference data generation, we propose an iterative pairwise ranking mechanism that derives preference ranking of completions using pairwise comparison signals. For training regularization, we observe that preference optimization tends to achieve better convergence when the LLM predicted likelihood of preferred samples gets slightly reduced.
  arXiv  Detail & Related papers  (2024-11-07T23:03:11Z)
• Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models [79.84205827056907]
  We present Self-Steering Optimization ($SSO$), an algorithm that autonomously generates high-quality preference data.<n>$SSO$ employs a specialized optimization objective to build a data generator from the policy model itself, which is used to produce accurate and on-policy data.<n>Our evaluation shows that $SSO$ consistently outperforms baselines in human preference alignment and reward optimization.
  arXiv  Detail & Related papers  (2024-10-22T16:04:03Z)
• An incremental preference elicitation-based approach to learning potentially non-monotonic preferences in multi-criteria sorting [53.36437745983783]
  We first construct a max-margin optimization-based model to model potentially non-monotonic preferences. We devise information amount measurement methods and question selection strategies to pinpoint the most informative alternative in each iteration. Two incremental preference elicitation-based algorithms are developed to learn potentially non-monotonic preferences.
  arXiv  Detail & Related papers  (2024-09-04T14:36:20Z)
• DsDm: Model-Aware Dataset Selection with Datamodels [81.01744199870043]
  Standard practice is to filter for examples that match human notions of data quality. We find that selecting according to similarity with "high quality" data sources may not increase (and can even hurt) performance compared to randomly selecting data. Our framework avoids handpicked notions of data quality, and instead models explicitly how the learning process uses train datapoints to predict on the target tasks.
  arXiv  Detail & Related papers  (2024-01-23T17:22:00Z)
• Functional Graphical Models: Structure Enables Offline Data-Driven Optimization [111.28605744661638]
  We show how structure can enable sample-efficient data-driven optimization. We also present a data-driven optimization algorithm that infers the FGM structure itself.
  arXiv  Detail & Related papers  (2024-01-08T22:33:14Z)
• PASTA: Pessimistic Assortment Optimization [25.51792135903357]
  We consider a class of assortment optimization problems in an offline data-driven setting. We propose an algorithm referred to as Pessimistic ASsortment opTimizAtion (PASTA) based on the principle of pessimism.
  arXiv  Detail & Related papers  (2023-02-08T01:11:51Z)
• Integrating Material Selection with Design Optimization via Neural Networks [6.114822256728633]
  In this paper, we propose the use of variational autoencoders (VAE) for simultaneous optimization. The proposed framework is demonstrated using trusses, where an optimal material needs to be chosen from a database. Several numerical examples illustrate the efficacy of the proposed framework.
  arXiv  Detail & Related papers  (2021-12-23T14:28:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.