GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training

• URL: http://arxiv.org/abs/2605.26184v1
• Date: Mon, 25 May 2026 07:52:29 GMT
• Title: GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training
• Abstract summary: We propose GAC, a noise-aware controller that derives an adaptive mixing weight from online estimates of gradient variance and disagreement between the two training signals.<n>Experiments on math, code, science, and logic benchmarks show that GAC consistently improves hybrid post-training over strong fixed and rule-based baselines.
• Score: 21.974153439592317
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hybrid post-training usually combines supervised fine-tuning and reinforcement learning, but fixed mixing schedules cannot adapt when the relative noise of the two signals changes over time. We propose GAC, a noise-aware controller that derives an adaptive mixing weight from online estimates of gradient variance and disagreement between the two training signals. The method adds smoothing, prior guidance, and bounded updates while reusing existing training tensors. Experiments on math, code, science, and logic benchmarks show that GAC consistently improves hybrid post-training over strong fixed and rule-based baselines, with larger gains at larger model scales and less than 1% training overhead.

Related papers

• Train Smarter, Not Harder: Switching Signal-Guided Training in Active Learning [1.215621649989295]
  We show that retraining is most useful in early rounds, while fine-tuning becomes safer once the model trajectory stabilizes.<n>We propose HybridAL, an adaptive training schedule that monitors an online stabilization signal and switches from retraining to fine-tuning after sustained stabilization.
  arXiv  Detail & Related papers  (2026-09-06T19:45:17Z)
• Confusion-Geometry Rebalancing for Long-Tailed Adversarial Training [53.3522202888031]
  Adversarial training under long tailed distributions suffers from a dual imbalance: the class imbalance skews the training objective toward head classes.<n>We propose a Confusion Geometry Rebalancing method (CGRm) for long tail adversarial training, a plug in framework that leverages directed robust errors as training signals.
  arXiv  Detail & Related papers  (2026-08-10T14:54:00Z)
• NoiseRater: Meta-Learned Noise Valuation for Diffusion Model Training [81.58321000203712]
  NoiseRater is a meta-learning framework for instance-level noise valuation in diffusion model training.<n>NoiseRater assigns importance scores to individual noise realizations conditioned on data and timestep.<n>Experiments on FFHQ and ImageNet demonstrate that not all noise samples contribute equally.
  arXiv  Detail & Related papers  (2026-05-02T19:43:16Z)
• The Coupling Within: Flow Matching via Distilled Normalizing Flows [45.3372883861006]
  A crucial ingredient in flow training is the choice of coupling measure for sampling noise/data pairs.<n>We propose Normalized Flow Matching, a new method that distills the quasi-deterministic coupling of pretrained NF models to train student flow models.
  arXiv  Detail & Related papers  (2026-03-09T23:07:36Z)
• GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control [16.529035487811267]
  We show that naively applying asynchrony to policy-gradient updates can induce qualitatively different training dynamics and lead to severe training instability.<n>We propose GRADIENT ALIGNMENT CONTROL, a simple dynamics-aware stabilization method that regulates asynchronous RL progress along stale-aligned directions.
  arXiv  Detail & Related papers  (2026-03-02T06:19:43Z)
• Conda: Column-Normalized Adam for Training Large Language Models Faster [70.66067959375748]
  Column-Normalized Adam (Conda) is a novel approach to large language models (LLMs)<n>Conda projects updates into a subspace and applies column-wise second moment normalization based on the projected gradients.<n>Experiments on the LLaMA and GPT-2 series show that Conda consistently outperforms AdamW, Muon, and other baselines in pre-training.
  arXiv  Detail & Related papers  (2025-09-29T02:58:19Z)
• Learn2Mix: Training Neural Networks Using Adaptive Data Integration [14.314972805316202]
  learn2mix is a new training strategy that adaptively adjusts class proportions within batches, focusing on classes with higher error rates.<n> Empirical evaluations show that neural networks trained with learn2mix converge faster than those trained with existing approaches.
  arXiv  Detail & Related papers  (2024-12-21T04:40:07Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls [77.42510898755037]
  One More Step (OMS) is a compact network that incorporates an additional simple yet effective step during inference. OMS elevates image fidelity and harmonizes the dichotomy between training and inference, while preserving original model parameters. Once trained, various pre-trained diffusion models with the same latent domain can share the same OMS module.
  arXiv  Detail & Related papers  (2023-11-27T12:02:42Z)
• Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality [65.67315418971688]
  Nearest Orthogonal Gradient (NOG) and Optimal Learning Rate (OLR) are proposed. Experiments on visual recognition demonstrate that our methods can simultaneously improve the covariance conditioning and generalization.
  arXiv  Detail & Related papers  (2022-07-05T15:39:29Z)
• Self-Adaptive Training: beyond Empirical Risk Minimization [15.59721834388181]
  We propose a new training algorithm that dynamically corrects problematic labels by model predictions without incurring extra computational cost. Self-adaptive training significantly improves generalization over various levels of noises, and mitigates the overfitting issue in both natural and adversarial training. Experiments on CIFAR and ImageNet datasets verify the effectiveness of our approach in two applications.
  arXiv  Detail & Related papers  (2020-02-24T15:47:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.