Daily Paper Briefing

Abstract Overview

UNREAL is a model-native evidence selection framework that uses a single frozen decoder-only LLM for both corpus retrieval and long-context reasoning. The method encodes candidate chunks with the LLM, derives retrieval queries from internal residual-stream states, and learns only retrieval tokens and layer-mixing weights, adding fewer than 500K trainable parameters. The paper evaluates UNREAL on full-corpus Wikipedia retrieval over a 3B-token, 21M-chunk index and on long-context benchmarks where relevant evidence is sparse among many distractors. Across these settings, the authors demonstrate that retrieval and long-context inference can be unified as the same evidence-selection problem at different scales.

Novelty

The key novelty is that a single frozen LLM explicitly selects evidence across both corpus-scale retrieval and long-context prompting using its own internal representations. Unlike standard RAG pipelines that rely on an external retriever, UNREAL derives retrieval queries directly from internal residual states and applies uniformly across dense, linear-attention, and state-space hybrid backbones without backbone finetuning.

Results

On the 21M-chunk Wikipedia index, all four tested UNREAL backbones outperform reported dense and hybrid retriever-reranker baselines, raising recall@10 on HotpotQA from 49.1% to 73.2%, on 2WikiMultiHopQA from 31.7% to 60.1%, and on MuSiQue from 8.8% to 14.4%. On long-context benchmarks, the same mechanism increases NoLiMa accuracy from 1.0% to 24.83% at 128K tokens and LV-Eval F1 from 49.97% to 54.66% at 256K, while reducing FLOPs and time-to-first-token relative to full-context inference from roughly 32K tokens onward.

Key Points

1. UNREAL trains a lightweight retrieval head on top of a frozen decoder-only LLM, leveraging internal residual-stream representations to score and select evidence chunks.
2. The framework unifies full-corpus retrieval over 21M Wikipedia chunks and in-prompt long-context evidence selection within a single model architecture.
3. Empirical evaluations show higher retrieval accuracy and long-context QA performance than external retriever pipelines and full-context inference, with efficiency gains at long context lengths.

References

• arXiv: https://arxiv.org/abs/2610.08463v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2610.08463v1
• Hugging Face Papers: https://huggingface.co/papers/2610.08463