DIAG: Diagnostic Iterative Alignment and Generation for Data-Efficient Mathematical Preference Distillation

• URL: http://arxiv.org/abs/2608.22806v1
• Date: Mon, 24 Aug 2026 05:09:12 GMT
• Title: DIAG: Diagnostic Iterative Alignment and Generation for Data-Efficient Mathematical Preference Distillation
• Abstract summary: We propose a Diagnostic Iterative Alignment and Generation framework to increase informative supervision and focus training near the student's current competence boundary.<n>Experiments show that DIAG boosts yield across iterations and delivers stronger reasoning performance under an iso-effective training budget.
• Score: 7.6963470169750465
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Iterative preference optimization is essential for aligning Large Language Models on mathematical reasoning tasks, yet its efficiency is often throttled by signal scarcity: as the model improves, static problem sets become increasingly mismatched to the model's evolving competence, producing rollouts that are either too easy or too hard and therefore non-informative, which leads to a scarcity of valid preference pairs. We propose DIAG, a Diagnostic Iterative Alignment and Generation framework that adaptively reshapes the practice distribution to increase informative supervision and focus training near the student's current competence boundary. DIAG consists of two phases: (1) diagnosing valid preference-pair yield to calibrate the exploration-exploitation trade-off and allocate topic quotas via an Empirical Bayes shrinkage estimator, thereby prioritizing high-yield concepts; and (2) generating targeted practice, where a teacher synthesizes variants from the student's failure traces. We further provide a theoretical view interpreting DIAG as a teacher-mediated approximation to KL-regularized reweighting of the practice distribution toward the student's competence boundary, where valid preference-pair yield is maximized. Experiments show that DIAG boosts yield across iterations and delivers stronger reasoning performance under an iso-effective training budget, demonstrating that it can distill more informative preference supervision for mathematical reasoning.

Related papers

• INSPIRE: An Internalize-Then-Improve Approach for Example-Driven Mathematical Reasoning [61.60016491769943]
  In human mathematics education, example-based reasoning such as constructing counterexamples to test theorem boundaries reflects deep conceptual understanding.<n>We propose INSPIRE, an Internalize-Then-Improve approach combining Reference-Guided Student Internalization (RGSI) with a stage-wise preference training strategy.<n>Experiments across multiple model scales and families demonstrate consistent improvements, even surpassing larger open-source models.
  arXiv  Detail & Related papers  (2026-08-27T03:24:03Z)
• CARE: Competence-Aware Reward Shaping for Adaptive Reasoning Length in Video-MLLMs [50.189987475377656]
  We propose CARE, a competence-aware reward shaping framework for adaptive reasoning length optimization in multimodal reasoning.<n> CARE maintains a smoothed competence estimate via an exponential moving average of pass rates, and uses it to route training into progressive stages.<n>Experiments on multiple video reasoning and general video understanding benchmarks demonstrate that CARE consistently improves reasoning accuracy, stabilizes reinforcement learning, and significantly enhances token efficiency.
  arXiv  Detail & Related papers  (2026-06-18T08:28:26Z)
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• Dual-Teacher Distillation with Subnetwork Rectification for Black-Box Domain Adaptation [69.7036779439312]
  Black box domain adaptation represents a highly practical yet extremely challenging setting.<n>We propose a dual teacher distillation with subnetwork rectification (DDSR) model that exploits specific knowledge embedded in black box source models.<n>Our approach demonstrates consistent improvements over state of the art methods, including those using source data or models.
  arXiv  Detail & Related papers  (2026-03-24T07:54:19Z)
• CTPD: Cross Tokenizer Preference Distillation [19.4149691480574]
  Cross-Tokenizer Preference Distillation (CTPD) is the first unified framework for transferring human-aligned behavior between models with heterogeneous tokenizers.<n>CTPD introduces three key innovations: (1) Aligned Span Projection, which maps teacher and student to shared character-level spans for precise supervision transfer; (2) a cross-tokenizer adaptation of Token-level Importance Sampling (TIS-DPO) for improved credit assignment; and (3) a Teacher-Anchored Reference, allowing the student to directly leverage the teacher's preferences in a DPO-style objective.
  arXiv  Detail & Related papers  (2026-01-17T01:11:35Z)
• Did Models Sufficient Learn? Attribution-Guided Training via Subset-Selected Counterfactual Augmentation [61.248535801314375]
  Subset-Selected Counterfactual Augmentation (SS-CA)<n>We develop Counterfactual LIMA to identify minimal spatial region sets whose removal can selectively alter model predictions.<n>Experiments show that SS-CA improves generalization on in-distribution (ID) test data and achieves superior performance on out-of-distribution (OOD) benchmarks.
  arXiv  Detail & Related papers  (2025-11-15T08:39:22Z)
• Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories [58.988535279557546]
  We introduce textbf sycophancy Mitigation through Adaptive Reasoning Trajectories.<n>We show that SMART significantly reduces sycophantic behavior while preserving strong performance on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2025-09-20T17:09:14Z)
• Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning [10.186434946738201]
  Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.<n>Recent methods have improved reasoning through expanded corpus and multistage training combining reinforcement learning and supervised fine-tuning.
  arXiv  Detail & Related papers  (2025-08-13T15:32:25Z)
• Binarization-Aware Adjuster: Bridging Continuous Optimization and Binary Inference in Edge Detection [0.0]
  Image edge detection (ED) faces a fundamental mismatch between training and inference.<n>In this paper, we propose a theoretical method to design a Binarization-Aware (BAA)<n>BAA explicitly incorporates binarization behavior into gradient-based optimization.
  arXiv  Detail & Related papers  (2025-06-14T11:56:44Z)
• LARES: Latent Reasoning for Sequential Recommendation [96.26996622771593]
  We present LARES, a novel and scalable LAtent REasoning framework for Sequential recommendation.<n>Our proposed approach employs a recurrent architecture that allows flexible expansion of reasoning depth without increasing parameter complexity.<n>Our framework exhibits seamless compatibility with existing advanced models, further improving their recommendation performance.
  arXiv  Detail & Related papers  (2025-05-22T16:22:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.