ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
Abstract Overview
This paper presents ReContext, a training-free inference method for long-context reasoning with large language models. The method uses model-internal, question-conditioned relevance signals to identify candidate evidence in the original long context, materializes those signals into grounded text spans, and replays the selected evidence before final answer generation. Unlike retrieval, compression, or pruning approaches, it keeps the full original context available and uses the replayed evidence pool for emphasis rather than exclusion. The paper also gives an associative-memory interpretation of the method and a theoretical monotonic-improvement result for recursive evidence replay.
Novelty
The distinctive idea is to turn internal attention-based relevance signals into an explicit, recursively updated evidence pool that is replayed near the question while preserving the full prompt. The work is also unusual in pairing this inference-time mechanism with an associative-memory framing and a theorem arguing monotonic improvement of the hidden representation toward the answer embedding.
Results
Across eight 128K-context datasets and three backbones (Qwen3-4B, Qwen3-8B, and Llama3-8B), ReContext achieves the best average rank on all three model families and improves mean accuracy over Vanilla from 0.24 to 0.30, a reported 24.6% relative gain. It also remains competitive under a 64K context budget and with thinking enabled, and the reported efficiency analysis notes minimal memory overhead due to incorporating fewer than 128 extra evidence tokens.
Key Points
- ReContext separates evidence organization from answer generation by recursively selecting and replaying grounded evidence spans while keeping the complete original context accessible.
- The method relies on internal question-to-context relevance signals, avoiding external retrieval, prompt pruning, or backbone modifications during final decoding.
- Empirically, the method shows consistent aggregate gains across multiple long-context benchmarks, while ablations indicate that recursion depth, evidence budget, and selecting evidence from the original context all affect performance.