One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control

• URL: http://arxiv.org/abs/2609.06469v1
• Date: Sun, 06 Sep 2026 08:32:59 GMT
• Title: One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control
• Abstract summary: We show that same-point domain gradients may remain nearly even when consecutive realized updates partially reverse one another in output space.<n>We propose OSOL, which designates a focus domain at each iteration, to rank token-level rebound risk.<n>OSOL reaches a domain-macro average of 0.4822, improving by 5.7% over the strongest compared baseline, without explicit higher-order differentiation.
• Score: 62.82372407840088
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) across multiple domains can broaden the reasoning capabilities of large language models (LLMs), yet joint training often degrades individual-domain performance and can destabilize optimization. Existing work typically diagnoses such interference from a single-step view using first-order gradient alignment or curvature-based proxies. We show that this view can miss a critical form of sequential interference: same-point domain gradients may remain nearly orthogonal even when consecutive realized updates partially reverse one another in output space. We further show that consecutive token log-probability footprints recover this interaction directly from adjacent checkpoints as a local second-order interaction in output space, without explicitly reconstructing same-step curvature. Building on this insight, we propose OSOL, which designates a focus domain at each iteration, uses the preceding checkpoint footprint to rank token-level rebound risk, and applies a drift-ranked, adaptively scaled correction within the standard GRPO update. Our analysis shows that this correction suppresses the targeted cross-step output backtracking component. Controlled studies further show that cross-step backtracking is more strongly associated with subsequent task damage than same-point gradient diagnostics, while the preceding footprint ranks future rebound risk more accurately than Hessian-based proxies. On Qwen3-30B-A3B, OSOL reaches a domain-macro average of 0.4822, improving by 5.7% over the strongest compared baseline, without explicit higher-order differentiation.

Related papers

• Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning [7.885980895039013]
  We argue that the right resolution is state-dependent, and propose GACA, a critic-free estimator whose granularity follows an uncertainty-based criticality proxy.<n>GACA scores every step by the negative log-likelihood its own rollout already records, then blends the two advantages with a per-step weight that grows with that score.<n>On ALFWorld and WebShop, GACA improves task success over GRPO and GiGPO at both 1.5B and 7B scales.
  arXiv  Detail & Related papers  (2026-09-11T04:19:44Z)
• Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning [14.631835952085263]
  We propose a token-level masking rule that aligns updates with a finite-horizon policy-improvement bound via two mechanisms.<n> CPPO enhances training stability and significantly improves reasoning accuracy across various model scales.
  arXiv  Detail & Related papers  (2026-06-09T15:09:29Z)
• When Attribution Patching Lies: Diagnosis and a Second-Order Correction [9.338965648455238]
  We show that the dominant error stems from the non-linearities in the downstream network rather than local curvature at the patched component.<n>This insight yields three practical tools: (i) a reliability score to detect untrustworthy estimates, (ii) error bounds quantifying potential attribution mis-specifications, and (iii) a Hessian-vector-product correction that eliminates the leading-order error with only one additional backward pass.
  arXiv  Detail & Related papers  (2026-06-05T05:20:32Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• DISA: Offline Importance Sampling for Distribution-Matching LLM-RL [56.9445657766829]
  We introduce DISA, which moves this calibration problem outside the RL loop.<n> DISA draws proposal trajectories offline, estimates the partition function via importance sampling, and freezes the resulting partition-function estimate.<n>On two open-weight backbones across six math and three code benchmarks, DISA matches or exceeds the online-coupled distribution-matching baseline Flow.
  arXiv  Detail & Related papers  (2026-05-17T07:14:44Z)
• AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems [30.172832661173928]
  LLM-based multi-agent systems are increasingly deployed on long-horizon tasks.<n>A single decisive error is often accepted by downstream agents and cascades into trajectory-level failure.<n>We introduce AgentForesight, a framework that reframes this problem as online auditing.
  arXiv  Detail & Related papers  (2026-05-09T05:55:19Z)
• Convergence of Byzantine-Resilient Gradient Tracking via Probabilistic Edge Dropout [1.3902537392439644]
  We study distributed optimization over networks with Byzantine agents that may send arbitrary adversarial messages.<n>We propose emphGradient Tracking with Probabilistic Edge Dropout and Leaky Integration (GT-PD-L)<n>GT-PD-L outperforms coordinate-wise mean by up to 4.3 percentage points under stealth attacks.
  arXiv  Detail & Related papers  (2026-04-01T03:55:42Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction [58.51530390018909]
  Large Language Model based multi-agent systems excel at collaborative problem solving but remain brittle to cascading errors.<n>We present MASC, a metacognitive framework that endows MAS with real-time, unsupervised, step-level error detection and self-correction.
  arXiv  Detail & Related papers  (2025-10-16T05:35:37Z)
• BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models [57.304411396229035]
  We present BranchGRPO, a method that restructures the rollout process into a branching tree.<n>On HPDv2.1 image alignment, BranchGRPO improves alignment scores by up to textbf16% over DanceGRPO.<n>A hybrid variant, BranchGRPO-Mix, further accelerates training to 4.7x faster than DanceGRPO without degrading alignment.
  arXiv  Detail & Related papers  (2025-09-07T12:53:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.