Efficient Listwise Reranking with Compressed Document Representations

• URL: http://arxiv.org/abs/2604.26483v1
• Date: Wed, 29 Apr 2026 09:48:44 GMT
• Title: Efficient Listwise Reranking with Compressed Document Representations
• Abstract summary: We introduce RRK, an efficient listwise reranker compressing documents into multi-token fixed-size embedding representations.<n>Our simple training via distillation shows that this combination of rich compressed representations and listwise reranking yields a highly efficient and effective system.
• Score: 10.623994225850046
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reranking, the process of refining the output from a first-stage retriever, is often considered computationally expensive, especially when using Large Language Models (LLMs). A common approach to mitigate this cost involves utilizing smaller LLMs or controlling input length. Inspired by recent advances in document compression for retrieval-augmented generation (RAG), we introduce RRK, an efficient and effective listwise reranker compressing documents into multi-token fixed-size embedding representations. Our simple training via distillation shows that this combination of rich compressed representations and listwise reranking yields a highly efficient and effective system. In particular, our 8B-parameter model runs 3x-18x faster than smaller rerankers (0.6-4B parameters) while matching or outperforming them in effectiveness. The efficiency gains are even more striking on long-document benchmarks, where RRK widens its advantage further.

Related papers

• No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval [51.43543998583709]
  Single-stage Sparse Retrieval (SSR) is a paradigm shift that replaces expensive clustering with efficient sparse coding.<n>It reduces indexing time by 15x compared to ColBERTv2, halves retrieval latency, and simultaneously improves retrieval performance.
  arXiv  Detail & Related papers  (2026-05-28T15:53:34Z)
• Layer-wise Token Compression for Efficient Document Reranking [18.48737466474846]
  Cross-encoder rerankers suffer from high computational costs due to processing long query-document sequences at inference time.<n>We propose Layer-wise Token Compression, which applies adaptive token pooling at intermediate transformer layers.<n>We show that compression at middle layers preserves ranking quality while increasing inference QPS by up to 25% for passage ranking and up to 116% for document ranking.
  arXiv  Detail & Related papers  (2026-05-20T03:52:31Z)
• Very Efficient Listwise Multimodal Reranking for Long Documents [7.558933130117637]
  We propose ZipRerank, a highly efficient listwise multimodal reranker.<n>It reduces input length via a lightweight query-image early interaction mechanism.<n>Experiments on the MMDocIR benchmark show that ZipRerank matches or surpasses state-of-the-art multimodal rerankers.
  arXiv  Detail & Related papers  (2026-05-12T09:45:59Z)
• Gradually Compacting Large Language Models for Reasoning Like a Boiling Frog [72.4168434368873]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, but their substantial size often demands significant computational resources.<n>We propose a gradual compacting method that divides the compression process into multiple fine-grained iterations.<n>This iterative approach-reminiscent of the "boiling frog" effect-enables the model to be progressively compressed without abrupt performance loss.
  arXiv  Detail & Related papers  (2026-02-04T06:56:52Z)
• Efficiency Optimizations for Superblock-based Sparse Retrieval [3.8177503173791756]
  This paper proposes a simple and effective superblock pruning scheme that reduces the overhead of superblock score while preserving competitive relevance.<n>It combines this scheme with a compact index structure and a robust zero-shot configuration that is effective across LSR models and multiple datasets.
  arXiv  Detail & Related papers  (2026-02-02T22:42:22Z)
• SimpleMem: Efficient Lifelong Memory for LLM Agents [73.74399447715052]
  We introduce SimpleMem, an efficient memory framework based on semantic lossless compression.<n>We propose a three-stage pipeline designed to maximize information density and token utilization.<n> Experiments on benchmark datasets show that our method consistently outperforms baseline approaches in accuracy, retrieval efficiency, and inference cost.
  arXiv  Detail & Related papers  (2026-01-05T21:02:49Z)
• Memory-Efficient Fine-Tuning via Low-Rank Activation Compression [16.44044624606008]
  Low-Rank Activation Compression (LoRAct) is a memory-efficient fine-tuning approach.<n>LoRAct reduces activation memory by approximately 80% in comparison with the widely adopted LoRA method.
  arXiv  Detail & Related papers  (2025-09-27T19:48:32Z)
• READER: Retrieval-Assisted Drafter for Efficient LLM Inference [0.0386965802948046]
  Autoregressive Language Models instantiate a factorized likelihood over token sequences, yet their strictly sequential decoding process imposes an intrinsic lower bound on latency inference.<n>This bottleneck has emerged as a central obstacle to the scalable deployment of large-scale generative models.<n>We present READER, a speculative decoding framework that bypasses the training of the auxiliary draft model.
  arXiv  Detail & Related papers  (2025-08-12T16:47:48Z)
• Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers [58.98923344096319]
  REFORM is a novel inference framework that efficiently handles long contexts through a two-phase approach.<n>It achieves over 50% and 27% performance gains on RULER and BABILong respectively at 1M context length.<n>It also outperforms baselines on Infinite-Bench and MM-NIAH, demonstrating flexibility across diverse tasks and domains.
  arXiv  Detail & Related papers  (2025-06-01T23:49:14Z)
• Reranking with Compressed Document Representation [7.894770087349908]
  We reduce the input size by compressing documents into fixed-size embedding representations.<n>We then teach a reranker to use compressed inputs by distillation.<n>Although based on a billion-size model, our trained reranker using this compressed input can challenge smaller rerankers in terms of both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2025-05-21T11:35:11Z)
• Sliding Window Attention Training for Efficient Large Language Models [55.56483740523027]
  We introduce SWAT, which enables efficient long-context handling via Sliding Window Attention Training.<n>This paper first attributes the inefficiency of Transformers to the attention sink phenomenon.<n>We replace softmax with the sigmoid function and utilize a balanced ALiBi and Rotary Position Embedding for efficient information compression and retention.
  arXiv  Detail & Related papers  (2025-02-26T05:31:44Z)
• Efficient Long Context Language Model Retrieval with Compression [57.09163579304332]
  Long Context Language Models (LCLMs) have emerged as a new paradigm to perform Information Retrieval (IR)<n>We propose a new compression approach tailored for LCLM retrieval, which is trained to maximize the retrieval performance while minimizing the length of the compressed passages.<n>We show that CoLoR improves the retrieval performance by 6% while compressing the in-context size by a factor of 1.91.
  arXiv  Detail & Related papers  (2024-12-24T07:30:55Z)
• Static Pruning in Dense Retrieval using Matrix Decomposition [12.899105656025018]
  In the era of dense retrieval, document indexing and retrieval is largely based on encoding models that transform text documents into embeddings.<n>Recent studies have shown that it is possible to reduce embedding size without sacrificing - and in some cases improving - the retrieval effectiveness.<n>We present a novel static pruning method for reducing the dimensionality of embeddings using Principal Components Analysis.
  arXiv  Detail & Related papers  (2024-12-13T09:09:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.