LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards

• URL: http://arxiv.org/abs/2605.31584v1
• Date: Fri, 29 May 2026 17:51:40 GMT
• Title: LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards
• Abstract summary: Reinforcement learning with verifiable rewards (RLVR) has shown promise for this task.<n>Existing methods are limited by low-confusability distractors and sparse, outcome-only reward signals.<n>textscLongTraceRL produces training contexts that are far more challenging than those built by random sampling or one-shot search.
• Score: 53.339700196282905
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-context reasoning remains a central challenge for large language models, which often fail to locate and integrate key information in extensive distracting content. Reinforcement learning with verifiable rewards (RLVR) has shown promise for this task, yet existing methods are limited by low-confusability distractors and sparse, outcome-only reward signals that cannot supervise intermediate reasoning steps. To address these issues, we introduce \textsc{LongTraceRL}. For data construction, we generate multi-hop questions via knowledge graph random walks and leverage search agent trajectories to build \emph{tiered distractors}: documents the agent read but did not cite (high confusability) and documents that appeared in search results but were never opened (low confusability), producing training contexts that are far more challenging than those built by random sampling or one-shot search. For reward design, we propose a \emph{rubric reward} that uses the gold entities along each reasoning chain as fine-grained, entity-level process supervision. This rubric reward is applied only to responses with correct final answers (positive-only strategy), distinguishing the reasoning quality among correct responses and preventing reward hacking. Experiments on three reasoning LLMs (4B--30B) across five long-context benchmarks demonstrate that \textsc{LongTraceRL} consistently outperforms strong baselines and encourages comprehensive, evidence-grounded reasoning. Codes, datasets and models are available at \href{https://github.com/THU-KEG/LongTraceRL}{https://github.com/THU-KEG/LongTraceRL}.

Related papers

• GTA-RAG: Graph-Trajectory-Augmented Reinforcement Learning for Multi-Turn Retrieval-Augmented Reasoning [27.83191719981676]
  We present textscGTA-RAG, a graph-trajectory-augmented RL framework for retrieval-augmented reasoning.<n>Method consistently outperforms RL-based RAG baselines with both Qwen2.5-3B and Qwen2.5-7B backbones.
  arXiv  Detail & Related papers  (2026-08-23T16:05:20Z)
• Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning [42.17614767170884]
  We show that models extensively copy text from the input into their reasoning traces rather than productively solving the problem.<n>By separating each prompt into task-relevant key evidence and irrelevant distractor context, we show that the root cause is insufficient grounding.<n>Motivated by this diagnosis, we propose GEAR, a reward shaping method that augments the accuracy signal with a grounding reward for overlap with key evidence and a distractor penalty for overlap with irrelevant context.
  arXiv  Detail & Related papers  (2026-07-21T17:59:21Z)
• ExpRL: Exploratory RL for LLM Mid-Training [40.4311030968937]
  Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning.<n>We study a more automated approach: emphRL-based mid-training using large corpora of human-written question-answer data.<n>Our method, ExpRL, uses them as emphreward scaffolds: references are hidden from the policy and used only to construct problem-specific grading rubrics.
  arXiv  Detail & Related papers  (2026-06-15T17:50:44Z)
• LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards [51.45138356629732]
  We introduce LongRLVR to augment the sparse answer reward with a dense and verifiable context reward.<n>This auxiliary signal directly incentivizes the model for selecting the correct grounding information.<n>LongRLVR consistently and significantly outperforms the standard RLVR across all models and benchmarks.
  arXiv  Detail & Related papers  (2026-03-02T18:07:53Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards [60.0970117192627]
  Reinforcement learning (RL) has emerged as a critical technique for enhancing LLM-based deep search agents.<n>Existing approaches primarily rely on binary outcome rewards, which fail to capture the comprehensiveness and factuality of agents' reasoning process.<n>We propose textbfCitation-aware RL Rewards (CaRR), a fine-grained reward framework for deep search agents.
  arXiv  Detail & Related papers  (2026-01-09T18:57:53Z)
• Lightweight Latent Reasoning for Narrative Tasks [89.94576985780549]
  Large language models (LLMs) tackle complex tasks by generating long chains of thought or "reasoning traces"<n>We propose LiteReason, a latent reasoning method that can be interleaved with standard token sampling and easily combined with reinforcement learning.<n> LiteReason employs a lightweight Reasoning Projector module, trained to produce continuous latent tokens that help the model'skip' reasoning steps.
  arXiv  Detail & Related papers  (2025-12-01T22:07:32Z)
• InfoFlow: Reinforcing Search Agent Via Reward Density Optimization [37.266452141225415]
  Reinforcement Learning with Verifiable Rewards (RLVR) is a promising approach for enhancing agentic deep search.<n>In this paper, we formalize this challenge as the textbfReward Density Optimization problem, which aims to improve the reward obtained per unit of exploration cost.<n>We introduce textbfInfoFlow, a systematic framework that tackles this problem from three aspects.
  arXiv  Detail & Related papers  (2025-10-30T15:03:21Z)
• Reinforced Latent Reasoning for LLM-based Recommendation [92.56166822197919]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities in complex problem-solving tasks.<n>Existing methods typically rely on fine-tuning with explicit chain-of-thought (CoT) data.<n>In this work, we explore an alternative approach that shifts from explicit CoT reasoning to compact, information-dense latent reasoning.
  arXiv  Detail & Related papers  (2025-05-25T11:03:45Z)
• FLARE: Faithful Logic-Aided Reasoning and Exploration [47.46564769245296]
  We introduce a novel approach for traversing the problem space using task decompositions.<n>We use the Large Language Models to plan a solution, soft-formalise the query into facts and predicates using a logic programming code.<n>Our method allows us to compute the faithfulness of the reasoning process w.r.t. the generated code and analyse the steps of the multi-hop search without relying on external solvers.
  arXiv  Detail & Related papers  (2024-10-14T19:39:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.