Improving Long-Context Retrieval with Multi-Prefix Embedding

• URL: http://arxiv.org/abs/2606.23642v1
• Date: Mon, 22 Jun 2026 17:31:02 GMT
• Title: Improving Long-Context Retrieval with Multi-Prefix Embedding
• Abstract summary: Multi-Prefix Embedding partitions a document into chunks separated by EOS tokens, encodes the full sequence in a single causal forward pass, and extracts one embedding at each prefix boundary.<n>MPE retains cross-chunk context, enables chunk-level MaxSim matching, and trains with only document-level relevance labels.
• Score: 64.08968715893504
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-context retrieval exposes a tension: single-vector embeddings lose fine-grained detail, while token-level multi-vector methods incur prohibitive storage. We propose Multi-Prefix Embedding (MPE), which partitions a document into chunks separated by EOS tokens, encodes the full sequence in a single causal forward pass, and extracts one embedding at each prefix boundary. MPE retains cross-chunk context, enables chunk-level MaxSim matching, and trains with only document-level relevance labels. Experiments on MLDR-en, BrowseComp-Plus, and LongEmbed show that MPE is competitive with or outperforms single-vector, independent-chunk, and multi-vector baselines, while providing a natural source attribution mechanism for locating evidence chunks.

Related papers

• REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling [0.0]
  REIGN is a bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network (GN) rather than on raw tokens.<n>We release a synthetic long-document retrieval benchmark for contrastive training and evaluation at long context lengths.<n>A paired significance test puts it on par with models 1.6-4.3x larger on the patent task, and it stays within 0.65 nDCG@10 of a 20x-larger model on LoCo.
  arXiv  Detail & Related papers  (2026-08-30T16:50:38Z)
• UEmbed: Unified Sparse and Dense Multimodal Embeddings [56.1131203949042]
  We introduce UEmbed, a decoder-only multimodal embedding model.<n>It produces both sparse lexical and dense representations in one causal forward pass.<n>We train UEmbed at 2B, 4B, and 9B scales, and it outperforms multimodal embedding models trained on publicly available data.<n>Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.
  arXiv  Detail & Related papers  (2026-08-03T17:54:11Z)
• BitNet Text Embeddings [104.29781473344813]
  We present BITEMBED, an extreme low-bit framework for LLM-based text embedding that jointly targets encoding efficiency and vector storage.<n> BITEMBED converts pretrained LLM backbones into BitNet-style embedding encoders with ternary weights, quantized activations, and lightweight normalization refinement.<n> Experiments on MMTEB (eng) with Qwen3-0.6B and Gemma703-2M show that BITEMBED is largely comparable to full precision teacher embedders.
  arXiv  Detail & Related papers  (2026-06-24T10:37:01Z)
• Rethinking RAG in Long Videos: What to Retrieve and How to Use It? [56.38819694781005]
  We introduce V-RAGBench, a benchmark of $langle$query, evidence chunk, answer$rangle$ triplets that enables faithful, decoupled evaluation of retrieval and generation.<n>We also present CARVE, a method that runs parallel retrievers across configurations and employs chunk-adaptive reranking to identify the winning configuration for each chunk.
  arXiv  Detail & Related papers  (2026-06-11T10:05:49Z)
• M3DocDep: Multi-modal, Multi-page, Multi-document Dependency Chunking with Large Vision-Language Models [20.44093242908907]
  We propose M3DocDep, an LVLM-based pipeline that first recovers block-level dependencies and then constructs chunks along the recovered document tree.<n>Under a shared-block evaluation protocol, M3DocDep improves STEDS by +28.5 to +39.6 percent on DHP benchmarks, retrieval nDCG by +1.1 to +15.3 percent, and QA ANLS by +4.5 to +15.3 percent on corpus-level RAG benchmarks.
  arXiv  Detail & Related papers  (2026-04-17T05:33:45Z)
• Multi-Vector Index Compression in Any Modality [73.7330345057813]
  Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos.<n>We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC)<n>AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation.
  arXiv  Detail & Related papers  (2026-02-24T18:57:33Z)
• Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation [22.803751188961865]
  We argue retrieval should move beyond similarity matching and instead operate over latent components.<n>We propose xMemory, which builds a hierarchy of intact units and maintains a searchable high-level node organisation.
  arXiv  Detail & Related papers  (2026-02-02T12:04:58Z)
• Read As Human: Compressing Context via Parallelizable Close Reading and Skimming [34.83776292069694]
  RAM (Read As HuMan) is a context compression framework that adopts an adaptive hybrid reading strategy.<n>Inspired by human reading behavior, RAM partitions the context into segments and encodes them with the input query in parallel.<n> Experiments demonstrate that RAM outperforms existing baselines on multiple question answering and summarization benchmarks.
  arXiv  Detail & Related papers  (2026-02-02T09:10:56Z)
• Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings [52.49524240846879]
  We propose Hierarchical Token Prepending to mitigate attention-level compression and readout-level over-squashing.<n>HTP partitions the input into blocks and prepends block-level summary tokens to subsequent blocks, creating pathways for backward information flow.<n>As a simple, architecture-agnostic method, HTP enhances both zero-shot and finetuned models, offering a scalable route to superior long-document embeddings.
  arXiv  Detail & Related papers  (2025-11-18T19:37:40Z)
• Zero-shot Multimodal Document Retrieval via Cross-modal Question Generation [47.714317480436215]
  PREMIR is a simple framework that leverages the broad knowledge of an MLLM to generate cross modal pre questions (preQs) before retrieval.<n> Experiments show that PREMIR achieves state of the art performance on out of distribution benchmarks, including closed domain and multilingual settings.
  arXiv  Detail & Related papers  (2025-08-23T16:14:41Z)
• FragRel: Exploiting Fragment-level Relations in the External Memory of Large Language Models [54.13671100638092]
  We propose a fragment-connected Hierarchical Memory based Large Language Models (LLMs) We formulate the fragment-level relations in external memory and present several instantiations for different text types. We validate the benefits of involving these relations on long story understanding, repository-level code generation, and long-term chatting.
  arXiv  Detail & Related papers  (2024-06-05T09:31:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.