Beyond Verified Answers: Solver-Informed Self-Distillation for Bootstrapping Operations Research Language Models

• URL: http://arxiv.org/abs/2609.09957v1
• Date: Wed, 09 Sep 2026 09:46:45 GMT
• Title: Beyond Verified Answers: Solver-Informed Self-Distillation for Bootstrapping Operations Research Language Models
• Abstract summary: Large language models can translate natural-language descriptions into operations research (OR) formulations.<n>Training commonly relies on synthetic formulations validated by human experts or stronger models.<n>We propose SOLID: solver-artifact feedback generated by its own rollouts, making self-distillation a practical, evaluator-free source of dense supervision.
• Score: 12.289065714096262
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Modern large language models (LLMs) can translate natural-language descriptions into operations research (OR) formulations. Post-training techniques including reinforcement learning and on-policy self-distillation have further improved this capability. However, three limitations remain in training LLMs for OR formulations. First, training commonly relies on synthetic formulations validated by human experts or stronger models, constraining scalable supervision. Second, credit assignment is either coarse or costly: outcome rewards score an entire trajectory without locating the responsible modeling decision, whereas process-level supervision requires an additional evaluator. Third, privileged self-distillation can induce style mismatch by using solver context unavailable at deployment. We find that a model can improve from solver-artifact feedback generated by its own rollouts, making self-distillation a practical, evaluator-free source of dense supervision. Therefore, we propose SOLID: Solver-Informed On-Policy LearnIng through Self-Distillation, a novel framework for self-improving OR language models without verified answers or external evaluators. SOLID executes candidate programs from multiple rollouts, clusters their objectives, and selects a majority-group artifact as a pseudo-reference. The model then performs updates using group-relative advantages and dense self-supervision signals. Across multiple OR benchmarks, SOLID improves solution accuracy for both general-purpose and OR-tuned models over outcome-only group-relative training. These results show that solver artifacts can support scalable self-improvement without trusted answers.

Related papers

• RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learning [63.432969627395686]
  RefineRL is a novel approach designed to unleash the self-refinement capabilities of large language models for competitive programming problems.<n>Skeptical-Agent is an iterative self-refinement agent equipped with local execution tools to validate generated solutions against public test cases of CP problems.<n>A reinforcement learning solution incentivizes LLMs to self-refine with only standard RLVR data.
  arXiv  Detail & Related papers  (2026-04-01T11:54:57Z)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding [54.72617309922891]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a core paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs)<n>Previous practice requires the LLM to sequentially generate solutions and self-verifications using two separate prompt templates, which significantly reduces efficiency.<n>We propose LaSeR (Reinforcement Learning with Last-Token Self-Rewarding), an algorithm that simply augments the original RLVR loss with a MSE loss.
  arXiv  Detail & Related papers  (2025-10-16T17:55:11Z)
• Learning to Refine: Self-Refinement of Parallel Reasoning in LLMs [102.48588475875749]
  We introduce Generative Self-Refinement (GSR), a novel parallel test-time scaling framework.<n>GSR generates a set of candidate responses in parallel and then performs self-refinement to synthesize a new superior solution.<n>We show that our method achieves state-of-the-art performance across five mathematical benchmarks.
  arXiv  Detail & Related papers  (2025-08-27T06:51:48Z)
• Post-Training Large Language Models via Reinforcement Learning from Self-Feedback [3.73824942136665]
  Large Language Models (LLMs) often produce plausible but poorly-calibrated answers.<n>We present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that uses the model's own confidence as an intrinsic reward.
  arXiv  Detail & Related papers  (2025-07-29T15:46:26Z)
• Incentivizing LLMs to Self-Verify Their Answers [22.387551134333084]
  We propose a framework that incentivizes Large Language Models to self-verify their own answers.<n>We train our self-verification models based on Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B.<n> Experiments on multiple mathematical reasoning benchmarks show that our models can not only improve post-training performance but also enable effective test-time scaling.
  arXiv  Detail & Related papers  (2025-06-02T06:54:29Z)
• Self-rewarding correction for mathematical reasoning [19.480508580498103]
  We study self-rewarding reasoning large language models (LLMs)<n>LLMs can simultaneously generate step-by-step reasoning and evaluate the correctness of their outputs during the inference time-without external feedback.<n>We propose a two-staged algorithmic framework for constructing self-rewarding reasoning models using only self-generated data.
  arXiv  Detail & Related papers  (2025-02-26T23:01:16Z)
• Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains [114.76612918465948]
  Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data.<n>We propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models.
  arXiv  Detail & Related papers  (2025-01-10T04:35:46Z)
• Recursive Introspection: Teaching Language Model Agents How to Self-Improve [30.086494067593268]
  We develop RISE: Recursive IntroSpEction, an approach for fine-tuning large language models. Our experiments show that RISE enables Llama2, Llama3, and Mistral models to improve themselves with more turns on math reasoning tasks.
  arXiv  Detail & Related papers  (2024-07-25T17:35:59Z)
• LLMs can learn self-restraint through iterative self-reflection [57.26854891567574]
  Large Language Models (LLMs) must be capable of dynamically adapting their behavior based on their level of knowledge and uncertainty associated with specific topics. This adaptive behavior, which we refer to as self-restraint, is non-trivial to teach. We devise a utility function that can encourage the model to produce responses only when it is confident in them.
  arXiv  Detail & Related papers  (2024-05-15T13:35:43Z)
• SALMON: Self-Alignment with Instructable Reward Models [80.83323636730341]
  This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision. We develop an AI assistant named Dromedary-2 with only 6 exemplars for in-context learning and 31 human-defined principles.
  arXiv  Detail & Related papers  (2023-10-09T17:56:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.