SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models

• URL: http://arxiv.org/abs/2606.07705v1
• Date: Fri, 05 Jun 2026 10:00:19 GMT
• Title: SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models
• Abstract summary: We propose Stage-Aware Dynamic Weighting (SAW) as a lightweight, algorithm-agnostic dynamic weighting mechanism.<n>SAW reweights each dimension's reward or advantage contribution by its relative informativeness within the batch.<n> Experiments on tool-calling and text summarization tasks demonstrate that SAW consistently improves both training efficiency and final performance.
• Score: 66.92079232778707
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although multi-objective reinforcement learning (MORL) is central to aligning large language models with complex human preferences, the prevailing practice of static weighted summation overlooks a more fundamental phenomenon: reward learning is markedly asynchronous across objectives. Well-learned dimensions quickly produce homogeneous, low-variance signals whose residual noise contaminates the aggregated reward (in GRPO) or occupies a fixed share of the advantage budget (in GDPO), interfering with the scarce yet high-value signals carried by under-learned dimensions. To address this asynchrony, we propose Stage-Aware Dynamic Weighting (SAW), a lightweight, algorithm-agnostic dynamic weighting mechanism. SAW utilizes the coefficient of variation (CV) as a scale-invariant proxy for real-time informativeness, reweighting each dimension's reward or advantage contribution by its relative informativeness within the batch. Unlike gradient-based methods that require multiple forward and backward passes, SAW relies solely on batch-level statistics, introducing nearly negligible computational overhead. Experiments on tool-calling and text summarization tasks demonstrate that SAW consistently improves both training efficiency and final performance under both GRPO and GDPO frameworks, confirming it as a general-purpose plug-in for multi-reward LLM alignment. Our code is available at https://github.com/Zhaolutuan/SAW

Related papers

• ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers [60.09590321091875]
  Post-Training Quantization (PTQ) offers an attractive solution by compressing models using a small calibration set with minimal training overhead.<n>Most existing PTQ works adopt a static quantization paradigm that is uniformly applied to all instances.<n>We propose ScalePredictor, a dynamic quantization framework for accurate and efficient quantization scale learning of ViTs.
  arXiv  Detail & Related papers  (2026-06-20T08:33:55Z)
• SMI: Efficient Self-Supervised Learning via Mutual-Information-Inspired Dependency Optimization [8.864968998286772]
  Self-supervised learning (SSL) has achieved remarkable representation learning performance.<n>Many existing methods rely on large batch sizes, memory banks, momentum encoders, or global synchronization mechanisms.<n>We propose Semantic Mutual Information (SMI), a lightweight self-supervised objective derived from a mutual-information-inspired dependency formulation.
  arXiv  Detail & Related papers  (2026-06-06T20:50:04Z)
• Divide-and-Conquer Inference for Large-Scale Visual Recognition with Multimodal Large Language Models [12.459927405623624]
  Divide-and-Conquer Inference (DCI) is a novel test-time scaling strategy for visual recognition with MLLMs.<n>DCI decomposes complex global classification tasks into simpler, localized subproblems and employs a dynamic pruning mechanism to compress the search space.<n>As a model-agnostic, plug-and-play paradigm, DCI offers an efficient approach for scaling the inferential precision of MLLMs in large-scale scenarios.
  arXiv  Detail & Related papers  (2026-05-24T01:07:05Z)
• Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping [66.25536973294726]
  Post-training methods for text-to-image (T2I) models are prone to reward hacking.<n>Super-Linear Advantage Shaping (SLAS) reshapes the local policy space.<n>SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T17:59:25Z)
• Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective [85.06838178922791]
  Reinforcement Learning (RL) has proven highly effective for autoregressive language models.<n>But adapting these methods to diffusion large language models (dLLMs) presents fundamental challenges.<n>We propose a principled RL framework that treats entire sequence generation as a single action and uses the ELBO as a tractable sequence-level likelihood proxy.
  arXiv  Detail & Related papers  (2025-12-03T13:05:32Z)
• Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach [78.4812458793128]
  We propose textbfTACO, a test-time-scaling framework that applies a lightweight pseudo-count estimator as a high-fidelity verifier of action chunks.<n>Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits.
  arXiv  Detail & Related papers  (2025-12-02T14:42:54Z)
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs [78.09559830840595]
  We present the first systematic study on quantizing diffusion-based language models.<n>We identify the presence of activation outliers, characterized by abnormally large activation values.<n>We implement state-of-the-art PTQ methods and conduct a comprehensive evaluation.
  arXiv  Detail & Related papers  (2025-08-20T17:59:51Z)
• A General Adaptive Dual-level Weighting Mechanism for Remote Sensing Pansharpening [11.791358860917189]
  deep learning methods for remote sensing pansharpening have advanced rapidly.<n>Many existing methods struggle to fully leverage feature heterogeneity and redundancy.<n>We introduce a general adaptive dual-level weighting mechanism (ADWM) to address these challenges.
  arXiv  Detail & Related papers  (2025-03-17T14:24:00Z)
• One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments [43.107261545706415]
  Large Language Models (LLMs) have advanced rapidly but face significant memory demands. Current methods typically require lengthy training to alleviate the performance degradation from quantization loss. We make an initial attempt to extend the once-for-all framework to large language models.
  arXiv  Detail & Related papers  (2024-05-30T16:05:15Z)
• Unifying Synergies between Self-supervised Learning and Dynamic Computation [53.66628188936682]
  We present a novel perspective on the interplay between SSL and DC paradigms. We show that it is feasible to simultaneously learn a dense and gated sub-network from scratch in a SSL setting. The co-evolution during pre-training of both dense and gated encoder offers a good accuracy-efficiency trade-off.
  arXiv  Detail & Related papers  (2023-01-22T17:12:58Z)
• Dynamic Multi-Scale Loss Optimization for Object Detection [14.256807110937622]
  We study the objective imbalance of multi-scale detector training. We propose an Adaptive Variance Weighting (AVW) to balance multi-scale loss according to the statistical variance. We develop a novel Reinforcement Learning Optimization (RLO) to decide the weighting scheme probabilistically during training.
  arXiv  Detail & Related papers  (2021-08-09T13:12:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.