Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents

• URL: http://arxiv.org/abs/2603.01481v1
• Date: Mon, 02 Mar 2026 05:44:50 GMT
• Title: Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents
• Abstract summary: We propose a framework that disentangles optimization across time scales.<n>Experiments with a high-fidelity user simulator show DuCA outperforms the state-of-the-art GRPO baseline.
• Score: 29.076822653598885
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Optimizing large language models for industrial sales requires balancing long-term commercial objectives (e.g., conversion rate) with immediate linguistic constraints such as fluency and compliance. Conventional reinforcement learning often merges these heterogeneous goals into a single reward, causing high-magnitude session-level rewards to overwhelm subtler turn-level signals, which leads to unstable training or reward hacking. To address this issue, we propose Dual-Horizon Credit Assignment (DuCA), a framework that disentangles optimization across time scales. Its core, Horizon-Independent Advantage Normalization (HIAN), separately normalizes advantages from turn-level and session-level rewards before fusion, ensuring balanced gradient contributions from both immediate and long-term objectives to the policy update. Extensive experiments with a high-fidelity user simulator show DuCA outperforms the state-of-the-art GRPO baseline, achieving a 6.82% relative improvement in conversion rate, reducing inter-sentence repetition by 82.28%, and lowering identity detection rate by 27.35%, indicating a substantial improvement for an industrial sales scenario that effectively balances the dual demands of strategic performance and naturalistic language generation.

Related papers

• Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning [85.02566758103008]
  Group-Graph Policy Optimization (G2PO) is a novel group-based Reinforcement Learning algorithm tailored for multi-turn agentic tasks.<n>G2PO substantially outperforms state-of-the-art prompt-based and RL baselines, achieving remarkable success rate improvements of up to 22.2% over GRPO.
  arXiv  Detail & Related papers  (2026-06-22T08:12:47Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation [33.370957547486775]
  Granularity-adaptivE Advantage Reweighting reshapes the trajectory-level GRPO advantage using token- and segment-level signals.<n>GEAR consistently outperforms standard GRPO, self-distillation-only baselines, and token- or turn-level credit-assignment methods.
  arXiv  Detail & Related papers  (2026-05-12T09:38:38Z)
• Milestone-Guided Policy Learning for Long-Horizon Language Agents [67.87164829944238]
  We introduce a milestone-guided policy learning framework, BEACON, to train long-horizon language agents.<n>On ALFWorld, WebShop, and ScienceWorld, BEACON consistently outperforms GRPO and GiGPO.<n>These results establish milestone-anchored credit assignment as an effective paradigm for training long-horizon language agents.
  arXiv  Detail & Related papers  (2026-05-07T12:00:40Z)
• Learning to Credit the Right Steps: Objective-aware Process Optimization for Visual Generation [57.61006679699323]
  Reinforcement learning has emerged as an effective framework for post-training visual generative models with human preference signals.<n>In modern visual generation, multiple reward models are often used to capture heterogeneous objectives, such as visual quality, motion consistency, and text alignment.<n>We propose Objective-aware Trajectory Credit Assignment (OTCA), a structured framework for fine-grained GRPO training.
  arXiv  Detail & Related papers  (2026-04-21T08:37:13Z)
• VALOR: Value-Aware Revenue Uplift Modeling with Treatment-Gated Representation for B2B Sales [0.0]
  B2B sales organizations must identify "persuadable" accounts within zero-inflated revenue distributions to optimize expensive human resource allocation.<n>Standard uplift frameworks struggle with treatment signal collapse in high-dimensional spaces and a misalignment between regression calibration and the ranking of high-value "whales"<n>We introduce VALOR, a unified framework featuring a Treatment-Gated Sparse-Revenue Network that uses bilinear interaction to prevent causal signal collapse.
  arXiv  Detail & Related papers  (2026-04-02T19:08:23Z)
• Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing [79.88256756334327]
  Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision.<n>We propose Sample-Routed Policy Optimization (SRPO)<n>SRPO routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction.
  arXiv  Detail & Related papers  (2026-04-02T17:29:18Z)
• Rewards as Labels: Revisiting RLVR from a Classification Perspective [10.774056916074441]
  Reinforcement Learning with Verifiable Rewards has recently advanced the capabilities of Large Language Models in complex reasoning tasks.<n>We propose Rewards as Labels (REAL), a novel framework that revisits verifiable rewards as categorical labels rather than scalar weights.<n>Our analysis reveals that REAL induces a monotonic and bounded gradient weighting, enabling balanced gradient allocation across rollouts.
  arXiv  Detail & Related papers  (2026-02-05T13:11:36Z)
• The Enhanced Physics-Informed Kolmogorov-Arnold Networks: Applications of Newton's Laws in Financial Deep Reinforcement Learning (RL) Algorithms [1.2508796035825014]
  In finance, Deep Reinforcement Learning (DRL) is commonly used to generate discrete trade signals or to determine continuous portfolio allocations.<n>We propose a novel reinforcement learning framework that incorporates Physics-Informed Kolmogorov-Arnold Networks (PIKANs) into several DRL algorithms.<n>PIKAN-based agents consistently deliver higher cumulative and annualized returns, superior Sharpe and Calmar ratios, and more favorable drawdown characteristics.
  arXiv  Detail & Related papers  (2026-02-01T18:48:33Z)
• MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matching [60.886768806064936]
  Tool-Integrated Reasoning empowers large language models to tackle complex tasks by interleaving reasoning steps with external tool interactions.<n>Existing reinforcement learning methods rely on outcome- or trajectory-level rewards, assigning uniform advantages to all steps within a trajectory.<n>We propose MatchTIR, a framework that introduces fine-grained supervision via bipartite matching-based turn-level reward assignment and dual-level advantage estimation.
  arXiv  Detail & Related papers  (2026-01-15T18:59:23Z)
• Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning [60.00161035836637]
  Group Relative Policy Optimization has emerged as a promising critic-free reinforcement learning paradigm for reasoning tasks.<n>We introduce Outcome-grounded Advantage Reshaping (OAR), a fine-grained credit assignment mechanism that redistributes advantages based on how much each token influences the model's final answer.<n>OAR-G achieves comparable gains with negligible computational overhead, both significantly outperforming a strong GRPO baseline.
  arXiv  Detail & Related papers  (2026-01-12T10:48:02Z)
• Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment [61.80228667422234]
  VGPO redefines value estimation across both temporal and group dimensions.<n>It transforms the sparse terminal reward into dense, process-aware value estimates.<n>It replaces standard group normalization with a novel process enhanced by absolute values to maintain a stable optimization signal.
  arXiv  Detail & Related papers  (2025-12-13T16:31:26Z)
• Reflecting with Two Voices: A Co-Adaptive Dual-Strategy Framework for LLM-Based Agent Decision Making [24.534365665776672]
  Large language model (LLM) agents often rely on external demonstrations or retrieval-augmented planning.<n>We propose DuSAR - a demonstration-free framework that enables a single frozen LLM to perform co-adaptive reasoning.<n>On ALFWorld and Mind2Web, DuSAR achieves state-of-the-art performance with open-source LLMs.
  arXiv  Detail & Related papers  (2025-12-09T08:44:59Z)
• Enhancing Agentic RL with Progressive Reward Shaping and Value-based Sampling Policy Optimization [13.475938754147625]
  Large Language Models (LLMs) empowered with Tool-Integrated Reasoning (TIR) can iteratively plan, call external tools, and integrate returned information to solve complex, long-horizon reasoning tasks.<n>Agentic Reinforcement Learning (Agentic RL) optimize such models over full tool-interaction trajectories.<n>Two key challenges hinder effectiveness: (1) Sparse, non-instructive rewards, such as binary 0-1 verifiable signals, provide limited guidance for intermediate steps and slow convergence.<n>We propose two complementary techniques: Progressive Reward Shaping (PRS) and Value-based Sampling Policy Optimization (VSPO).
  arXiv  Detail & Related papers  (2025-12-08T11:59:25Z)
• DualGR: Generative Retrieval with Long and Short-Term Interests Modeling [23.123644321765607]
  Generative Retrieval (GR) has emerged as a viable alternative to Embedding-Based Retrieval (EBR)<n>We propose DualGR, a generative retrieval framework that explicitly models dual horizons of user interests with selective activation.<n>Online A/B testing shows +0.527% video views and +0.432% watch time lifts, validating DualGR as a practical and effective paradigm for industrial generative retrieval.
  arXiv  Detail & Related papers  (2025-11-16T09:20:54Z)
• Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling [90.87033586963828]
  Outcome-reward reinforcement learning (RL) is a common and increasingly significant way to refine the step-by-step reasoning of multimodal large language models (MLLMs)<n>We propose Self-Consistency Sampling (SCS) to correct this issue.<n>Based on Qwen2.5-VL-7B-Instruct, SCS improves accuracy by up to 7.7 percentage points on six multimodal benchmarks with negligible extra computation.
  arXiv  Detail & Related papers  (2025-11-13T18:59:57Z)
• Token-Level Inference-Time Alignment for Vision-Language Models [58.41370989069588]
  Vision-Language Models (VLMs) have become essential backbones of modern multimodal intelligence.<n>We present TITA, a lightweight framework that freezes the base VLM and instead trains a reward model to approximate its distribution.<n>During inference, implicit preference signals are extracted as log-probability ratios between the reward model and the target VLM, yielding dense autoregressive feedback.
  arXiv  Detail & Related papers  (2025-10-20T09:58:03Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning [53.894789613838654]
  We introduce SEED-Bench-R1, a benchmark with complex real-world videos requiring balanced perception and reasoning.<n>Using SEED-Bench-R1, we find that standard GRPO, while improving answer accuracy, often reduces logical coherence between reasoning steps and answers, with only a 57.9% consistency rate.<n>We propose GRPO-CARE, a consistency-aware RL framework optimizing both answer correctness and reasoning coherence without explicit supervision.
  arXiv  Detail & Related papers  (2025-06-19T08:49:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.