Related papers: Context-Aware RL for Agentic and Multimodal LLMs

Context-Aware RL for Agentic and Multimodal LLMs

URL: http://arxiv.org/abs/2606.17053v1
Date: Mon, 15 Jun 2026 17:59:28 GMT
Title: Context-Aware RL for Agentic and Multimodal LLMs
Abstract summary: We propose ContextRL, a context-aware reinforcement learning method.<n>It rewards the model for selecting the context that supports the query--answer pair.<n>It achieves average gains of +2.2% over standard GRPO on 5 long-horizon benchmarks, and +1.8% across 12 diverse visual question answering benchmarks.
Score: 45.19288531697759
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Large language models (LLMs) often fail when answering requires identifying a small but decisive piece of evidence within a long or complex context, such as a single line in a tool trace or a subtle detail in an image. We propose ContextRL, a context-aware reinforcement learning (RL) method that improves long-horizon reasoning and multimodal performance through an \emph{indirect} auxiliary objective. Instead of supervising only the final answer, ContextRL presents the model with a query, an answer, and two highly similar contexts, and rewards it for selecting the context that supports the query--answer pair, thereby encouraging fine-grained grounding. We construct contrastive context data in two domains: for coding agents, trajectories serve as contexts, yielding 1k pairs built via condition filtering; for multimodal reasoning, images serve as contexts, yielding 7K pairs built via generative editing and similarity search. ContextRL achieves average gains of +2.2% over standard GRPO on 5 long-horizon benchmarks, and +1.8% across 12 diverse visual question answering benchmarks. To disentangle the effect of the proposed objective from that of additional data, we compare against data-augmentation baselines that repurpose the same contrastive contexts as standard query--context--answer examples. These baselines provide little to no improvement, showing that the gains arise from the proposed context-selection objective rather than from the contrastive data alone.

Related papers

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning [78.94047086588078]
We propose Recursive Evidence as LLM Harness for Long-Context Reasoning.<n>It is a training-free inference method for improving long-context reasoning.<n> Experiments on eight long-context datasets with 128K context length show it consistently improves evidence utilization.
arXiv Detail & Related papers (2026-07-02T17:59:26Z)
Context Recycling for Long-Horizon LLM Inference [0.3655021726150367]
Large language models (LLMs) exhibit strong capabilities in short-context reasoning but degrade in performance over long conversational horizons.<n>We introduce ContextForge, a system for context recycling that maintains task-relevant information across turns.<n>System enables efficient reuse of prior computation without relying on full context replay.
arXiv Detail & Related papers (2026-05-01T04:45:08Z)
Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage [89.58253972744531]
Retrieval-augmented generation (RAG) systems combine document retrieval with a generative model to address complex information seeking tasks like report generation.<n>We investigate whether upstream retrieval metrics can serve as reliable early indicators of the final generated response's information coverage.<n>Our findings demonstrate strong correlations between coverage-based retrieval metrics and nugget coverage in generated responses at both topic and system levels.
arXiv Detail & Related papers (2026-03-09T18:20:20Z)
Context-Picker: Dynamic context selection using multi-stage reinforcement learning [4.539896456749749]
We introduce emphContext-Picker, a reasoning-aware framework for long-context question answering.<n>Context-Picker treats context selection as a decision-making process optimized via a human-inspired, two-stage reinforcement learning schedule.<n>Experiments on five long-context and multi-hop QA benchmarks demonstrate that Context-Picker significantly outperforms strong RAG baselines.
arXiv Detail & Related papers (2025-12-16T14:52:11Z)
Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimization [56.97588709890706]
LongMab-PO is a novel framework that generates high-quality and diverse responses for long-context modeling tasks.<n> Experimental results show that LongMab-PO significantly improves the diversity and quality of preference data pairs.
arXiv Detail & Related papers (2025-08-19T16:33:55Z)
Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-$k$ [13.173467492641485]
We present Adaptive-$k$, a simple and effective single-pass method that adaptively selects the number of passages based on the similarity scores between the query and the candidate passages.<n>It does not require model fine-tuning, extra LLM inferences or changes to existing retriever-reader pipelines.<n>It improves accuracy across five LCLMs and two embedding models, highlighting that dynamically adjusting context size leads to more efficient and accurate QA.
arXiv Detail & Related papers (2025-06-10T06:11:01Z)
Context-DPO: Aligning Language Models for Context-Faithfulness [80.62221491884353]
We propose the first alignment method specifically designed to enhance large language models' context-faithfulness.<n>By leveraging faithful and stubborn responses to questions with provided context from ConFiQA, our Context-DPO aligns LLMs through direct preference optimization.<n>Extensive experiments demonstrate that our Context-DPO significantly improves context-faithfulness, achieving 35% to 280% improvements on popular open-source models.
arXiv Detail & Related papers (2024-12-18T04:08:18Z)
Sufficient Context: A New Lens on Retrieval Augmented Generation Systems [19.238772793096473]
Augmenting LLMs with context leads to improved performance across many applications.<n>We develop a new notion of sufficient context, along with a method to classify instances that have enough information to answer the query.<n>By stratifying errors based on context sufficiency, we find that larger models with higher baseline performance excel at answering queries when the context is sufficient.
arXiv Detail & Related papers (2024-11-09T02:13:14Z)
RARe: Retrieval Augmented Retrieval with In-Context Examples [40.963703726988946]
We introduce a simple approach to enable retrievers to use in-context examples. RARE finetunes a pre-trained model with in-context examples whose query is semantically similar to the target query. We find RARe exhibits stronger out-of-domain generalization compared to models using queries without in-context examples.
arXiv Detail & Related papers (2024-10-26T05:46:20Z)
DoubleDipper: Improving Long-Context LLMs via Context Recycling [44.24067814871803]
We propose DoubleDipper, a novel In-Context-Learning method that automatically generates few-shot examples for long context QA tasks.<n>We apply our method on multiple Large Language Models and obtain substantial improvements.<n>Surprisingly, despite introducing only single-hop ICL examples, LLMs successfully generalize to multi-hop long-context QA.
arXiv Detail & Related papers (2024-06-19T15:28:29Z)
Query Expansion Using Contextual Clue Sampling with Language Models [69.51976926838232]
We propose a combination of an effective filtering strategy and fusion of the retrieved documents based on the generation probability of each context. Our lexical matching based approach achieves a similar top-5/top-20 retrieval accuracy and higher top-100 accuracy compared with the well-established dense retrieval model DPR. For end-to-end QA, the reader model also benefits from our method and achieves the highest Exact-Match score against several competitive baselines.
arXiv Detail & Related papers (2022-10-13T15:18:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.