Bilevel Optimization of Synthetic Trajectories for Multi-Turn LLM Fine-Tuning

• URL: http://arxiv.org/abs/2605.24743v1
• Date: Sat, 23 May 2026 21:34:59 GMT
• Title: Bilevel Optimization of Synthetic Trajectories for Multi-Turn LLM Fine-Tuning
• Abstract summary: We propose BOOST, a bilevel optimization framework for learning with multi-turn trajectories.<n>We show that BOOST upweights synthetic trajectories that align with the real data distribution and exhibit higher qualitative merit.
• Score: 27.869067884673157
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While LLMs excel at single-turn generation, they struggle with long-horizon, multi-turn interactions. Offline reinforcement learning (RL) offers a scalable approach, yet its performance hinges on the availability and quality of multi-turn trajectory data. A common remedy is to augment training with synthetic trajectories generated by LLMs or simulators, but synthetic data is highly heterogeneous in quality, and naively treating all trajectories as equally informative can degrade performance. We propose BOOST, a bilevel optimization framework where the inner level trains the LLM on reweighted data and the outer level trains a lightweight reweighting head on held-out real validation tasks, assigning continuous trajectory-level weights without requiring an external judge. To ground this approach, we derive a PAC-Bayesian bound revealing a three-way trade-off: synthetic data increases diversity but risks task-shift, while concentrating weight on high-quality trajectories improves empirical performance at the cost of effective sample size. Empirically, our method consistently outperforms multiple baselines. Analysis reveals it upweights synthetic trajectories that align with the real data distribution and exhibit higher qualitative merit.

Related papers

• Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning [7.431998098521794]
  We introduce the Holistic Data Scheduler (HDS), a novel online data mixing framework.<n>HDS formulates the data scheduling challenge as a reinforcement learning problem in a continuous control space.<n>It integrates a data-driven reward for quality, a loss-driven reward capturing inter-domain influence, and a model-driven reward based on weight norms.
  arXiv  Detail & Related papers  (2026-06-23T04:32:46Z)
• PLOT: Enhancing Preference Learning via Optimal Transport [28.079554847535107]
  We introduce PLOT, which enhances Preference Learning in fine-tuning-based alignment through a token-level loss derived from Optimal Transport.<n>PLOT aligns model outputs with human preferences while preserving the original distribution of Large Language Models.
  arXiv  Detail & Related papers  (2026-04-02T09:51:56Z)
• Guided by Trajectories: Repairing and Rewarding Tool-Use Trajectories for Tool-Integrated Reasoning [65.10602992874787]
  AutoTraj is a framework that automatically learns TIR by repairing and rewarding tool-use trajectories.<n> Experiments on real-world benchmarks demonstrate the effectiveness of AutoTraj.
  arXiv  Detail & Related papers  (2026-01-30T14:42:04Z)
• Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration [56.074760766965085]
  PRISM achieves a dynamics-aware framework that arbitrates data based on its degree of cognitive conflict with the model's existing knowledge.<n>Our findings suggest that disentangling data based on internal optimization regimes is crucial for scalable and robust agent alignment.
  arXiv  Detail & Related papers  (2026-01-12T05:43:20Z)
• Hierarchical Optimization via LLM-Guided Objective Evolution for Mobility-on-Demand Systems [9.979671028876464]
  We propose a novel framework that integrates large language model (LLM) with mathematical optimization in a dynamic hierarchical system.<n>Within this framework, LLM serves as a meta-optimizer, producing semantics that guide a low-level responsible for constraint enforcement and real-time decision execution.<n>Experiments based on both the New York and Chicago taxi datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-10-12T14:56:19Z)
• Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward [54.708851958671794]
  We propose a Data-Efficient Policy Optimization pipeline that combines optimized strategies for both offline and online data selection.<n>In offline phase, we curate a high-quality subset of training samples based on diversity, influence, and appropriate difficulty.<n>During online RLVR training, we introduce a sample-level explorability metric to dynamically filter samples with low exploration potential.
  arXiv  Detail & Related papers  (2025-09-01T10:04:20Z)
• Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models [79.2162092822111]
  We systematically evaluate reinforcement learning (RL) and control-based methods on a suite of navigation tasks.<n>We employ a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning.<n>Our results show that model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts.
  arXiv  Detail & Related papers  (2025-02-20T18:39:41Z)
• Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement [41.929860869084536]
  We introduce Condor, a novel two-stage synthetic data generation framework that incorporates World Knowledge Tree and Self-Reflection Refinement to produce high-quality SFT data at scale.<n>Our experimental results demonstrate that a base model fine-tuned on only 20K Condor-generated samples achieves superior performance compared to counterparts.
  arXiv  Detail & Related papers  (2025-01-21T16:44:12Z)
• Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning [71.2981957820888]
  We propose a novel Star-Agents framework, which automates the enhancement of data quality across datasets. The framework initially generates diverse instruction data with multiple LLM agents through a bespoke sampling method. The generated data undergo a rigorous evaluation using a dual-model method that assesses both difficulty and quality.
  arXiv  Detail & Related papers  (2024-11-21T02:30:53Z)
• CAFE: Learning to Condense Dataset by Aligning Features [72.99394941348757]
  We propose a novel scheme to Condense dataset by Aligning FEatures (CAFE) At the heart of our approach is an effective strategy to align features from the real and synthetic data across various scales. We validate the proposed CAFE across various datasets, and demonstrate that it generally outperforms the state of the art.
  arXiv  Detail & Related papers  (2022-03-03T05:58:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.