Adaptive Simulation Experiment for LLM Policy Optimization

• URL: http://arxiv.org/abs/2604.08779v1
• Date: Thu, 09 Apr 2026 21:29:42 GMT
• Title: Adaptive Simulation Experiment for LLM Policy Optimization
• Abstract summary: Large language models (LLMs) have significant potential to improve operational efficiency in operations management.<n> Deploying these models requires specifying a policy that governs response quality, shapes user experience, and influences operational value.<n>This research proposes a pairwise comparison-based adaptive simulation experiment for identifying the optimal policy from a finite set of candidates.
• Score: 10.11723645147155
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have significant potential to improve operational efficiency in operations management. Deploying these models requires specifying a policy that governs response quality, shapes user experience, and influences operational value. In this research, we treat LLMs as stochastic simulators and propose a pairwise comparison-based adaptive simulation experiment framework for identifying the optimal policy from a finite set of candidates. We consider two policy spaces: an unstructured space with no parametric assumption, and a structured space in which the data are generated from a preference model. For both settings, we characterize the fundamental data requirements for identifying the optimal policy with high probability. In the unstructured case, we derive a closed-form expression for the optimal sampling proportions, together with a clear operational interpretation. In the structured case, we formulate a regularized convex program to compute the optimal proportions. We then develop an adaptive experimental procedure, termed LLM-PO, for both policy spaces, and prove that it identifies the optimal policy with the desired statistical guarantee while asymptotically attaining the fundamental data requirements. Numerical experiments demonstrate that LLM-PO consistently outperforms benchmark methods and improves LLM performance.

Related papers

• How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics [65.67654005892469]
  We show that proper instance-dependent sampling can yield stronger ranking guarantees, while skewed on-policy sampling can induce excessive concentration under structured preferences.<n>We then analyze iterative alignment dynamics in which the learned policy feeds back into future sampling and reference policies.<n>Our theoretical insights extend to Direct Preference Optimization, indicating the phenomena we captured are common to a broader class of preference-alignment methods.
  arXiv  Detail & Related papers  (2026-02-12T17:11:08Z)
• Bootstrapping LLMs via Preference-Based Policy Optimization [11.796630967998544]
  bootstrapping large language models (LLMs) through preference-based policy optimization offers a promising direction for aligning model behavior with human preferences.<n>We propose a novel preference-based policy optimization framework that formulates the learning process as a min-max game between the main policy and a reward model.<n>Our approach consistently outperforms existing state-of-the-art preference optimization techniques.
  arXiv  Detail & Related papers  (2025-11-17T01:41:14Z)
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning [77.92320830700797]
  Reinforcement Learning has played a central role in enabling reasoning capabilities of Large Language Models.<n>We propose a tractable computational framework that tracks and leverages curvature information during policy updates.<n>The algorithm, Curvature-Aware Policy Optimization (CAPO), identifies samples that contribute to unstable updates and masks them out.
  arXiv  Detail & Related papers  (2025-10-01T12:29:32Z)
• Alignment of large language models with constrained learning [93.2264691508005]
  We study the problem of computing an optimal large language model (LLM) policy for a constrained alignment problem.<n>We employ Lagrangian duality to develop an iterative dual-based alignment method that alternates between updating the policy via Lagrangian and updating a dual variable via dual descent.
  arXiv  Detail & Related papers  (2025-05-26T01:04:56Z)
• RL-finetuning LLMs from on- and off-policy data with a single algorithm [53.70731390624718]
  We introduce a novel reinforcement learning algorithm (AGRO) for fine-tuning large-language models.<n>AGRO leverages the concept of generation consistency, which states that the optimal policy satisfies the notion of consistency across any possible generation of the model.<n>We derive algorithms that find optimal solutions via the sample-based policy gradient and provide theoretical guarantees on their convergence.
  arXiv  Detail & Related papers  (2025-03-25T12:52:38Z)
• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences.<n>To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model.<n>Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Towards Efficient Exact Optimization of Language Model Alignment [93.39181634597877]
  Direct preference optimization (DPO) was proposed to directly optimize the policy from preference data. We show that DPO derived based on the optimal solution of problem leads to a compromised mean-seeking approximation of the optimal solution in practice. We propose efficient exact optimization (EXO) of the alignment objective.
  arXiv  Detail & Related papers  (2024-02-01T18:51:54Z)
• Statistical Rejection Sampling Improves Preference Optimization [42.57245965632205]
  We introduce a novel approach to source preference data from the target optimal policy using rejection sampling. We also propose a unified framework that enhances the loss functions used in both Sequence Likelihood (SLiC) and Direct Preference Optimization (DPO) from a preference modeling standpoint.
  arXiv  Detail & Related papers  (2023-09-13T01:07:25Z)
• Sample Efficient Model-free Reinforcement Learning from LTL Specifications with Optimality Guarantees [17.69385864791265]
  We present a model-free Reinforcement Learning (RL) approach that efficiently learns an optimal policy for an unknown system. We also provide improved theoretical results on choosing the key parameters to ensure optimality.
  arXiv  Detail & Related papers  (2023-05-02T12:57:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.